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memorability of book CHARACTERISTICS: AN EXPERIMENTAL STUDY 

Delores K. Vaughan 

Introduction 

If library catalogs had not existed for hundreds of years, and it 
were suddenly realized that a device for recording the contents of li- 
brary collections was desirable, how might the designers of the projected 
catalog proceed to develop specifications for their product? 

The considerations necessary to catalog development are first, 
to define the functions a library catalog would be expected to serve, 
next, to establish performance requirements it should meet, and finally, 
to develop a design to meet these requirements. 

An expected function of a catalog is that it be a finding device-- 
that it provide its users with the means of identifying and locating 
desired materials represented in the catalog. Moreover, one expects 
that the catalog will be utilized as a finding device in at least two 
ways: (1) by a patron who wishes to locate any appropriate materials 

available in the library on a particular subject or by a particular 
author, and (2) by a patron who desires to locate a specific work which 
he knows to exist--possibly one with which he has had previous contact. 

Let us concentrate on the last instance- -the patron who, having 
had previous contact with a particular book, wants to locate it in the 
catalog. How can the catalog provide him with access to the desired book? 
At this point we observe that present library catalogs provide access to 
books through three primary means of approach, author, title, and sub 
ject headings. Usually, a patron searching for a particular book needs 
accurate knowledge of at least one of these three in order to locate the 
book in the catalog. From the patron's point of view, however, an ideal 
catalog would allow him to utilize any information he could supply about 
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a particular book. He would not be dependent solely upon the traditional 
approaches, and thus would have a better chance of locating the book he 
sought. 

The introduction into library technology of computerized infor- 
mation handling systems affords opportunities for increasing catalogs' 
efficiency as finding devices. Automated systems could allow multiple 
coordinate searching on a number of information fields, or approaches-- 
a procedure which is impractical, if not impossible, with alphabetically 
arranged card catalogs. Thus a library patron desiring to retrieve a 
particular book might thus be able to interrogate an automated catalog 
with whatever fragmentary information he could provide. 

Important questions facing catalog designers are (1) what kinds 
of information are library patrons likely to be able to recall about 
books they have previously seen, and (2) how useful would such information 
be in a retrieval system. Catalog use studies seem to be of little value 
in exploring these questions, since the information library patrons bring 
to present catalogs is conditioned by their expectation of catalog capa- 
bilities . 

An investigation of these questions has been conducted by faculty 
and students of The Graduate Library School, University of Chicago, sup- 
ported by the National Science Foundation. The present set of reports 
concerns the findings of an experimental study, the book -memory experiment, 
designed to measure certain impressions persons retain after a single 
examination of a book. The research reported here is designed to provide 
evidence of the relative memorability of various descriptive character- 
istics of books. Characteristics investigated include several now 
utilized in descriptive and subject cataloging as well as various charac- 
teristics not commonly thought to be significant either for catalog 
records or for retrieval strategies. The primary objective of this phase 
Qf research is to identify those characteristics of books which are often 
enough remembered to be of potential use as retrieval clues. Such knowl- 
edge could then play an important role in design studies of future 
catalogs. 
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The Book Memory Experiment 



Objective and Scope 

The book-memory experiment was designed to partially simulate 
the condition confronting a library patron who is considering attempting 
retrieval of a book with which he has had previous experience. Unless 
he has an accurate written citauion, a patron in such a position must 
probe his memory for whatever pertinent facts about the desired work he 
can recall. With present catalogs, however, he must ignore facts which 
he believes will not be useful in retrieving the book: A correct author, 

title, or subject heading will be his most useful information. The book- 
memory experiment attempted to eliminate this characteristic mental 
self-limitation in order to obtain a more accurate representation of the 
types of information about books that people tend to remember. 

Subjects participating in the book-memory experiment, after a 
brief experience with specific books, were tested on their memory of 
selected characteristics of those books. Participants' responses to 
questions regarding the books were evaluated, and evidence of the rela- 
tive memorability of the various characteristics was gathered. A final 
assessment of the relative value in retrieval systems of these charac- 
teristics cannot be made on the basis of demonstrated memorability alone 
but is dependent upon factors in addition to the memory data. However, 
the data gathered from the book-memory experiment and summarized herein 

is also o£ interest in its own right. 

Design and Procedure 

The book-memory experiment was designed by a research staff of 
Graduate Library School faculty and students with the consultant services 
of two academic psychologists. After a pilot study, the experimental 
phase was conducted at the University of Chicago during the summer and 
fall of 1966. Materials Involved in the experiment included a collection 
of books gathered for the purpose and various paper-and-pencil testing 
devices. A self-selected group of volunteers acted as subjects. Each 
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subject attended two sessions; during the first, he was given exposure 
to a number of books, and during the second was quizzed about the books 
he had seen previously. 

The experiment was conducted in an office that had been specially 
equipped for the purpose. Book stacks were installed, and desks were 
provided and placed so that subjects were unable to look at the book 
stacks while completing their responses during the testing session. 

The experimental book collection consisted of 180 books chosen 
from the psychology^ collection of the University of Chicago Library and 
borrowed from the library for the duration of the project. Occasionally 
the library requested the return of one or more books; however, these 
books were kept until any subjects who had used them had completed their 
testing session. Titles of returned books were subsequently not offered 
to new subjects for their consideration, since the books were no longer 
available. The experimental collection was not seriously diminished as 
a consequence of books' being returned; only about ten books were lost 
in this manner while the experiment was being conducted. The books used 
were selected to provide a variety such as might be found in an actual 
library collection, and an effort was made to provide books representing 
various form, content, and physical characteristics. The experimental 
collection was housed in a single seven-foot book stack, in which the 
books were arranged alphabetically by author. A second or auxiliary 
book stack, housing a random collection of the same type of books, was 
used in the experiment to simulate another physical environment during 
recognition testing. 

Subjects for the book-memory experiment were paid volunteers 
from the University community who responded to an advertisement posted 
on campus bulletin boards. There was no screening of applicants; all 



^The field of psychology was chosen in order to provide a con- 
trolled subject-matter collection, one in which there is a fairly high 
degree of lay interest, and one in which terminology is not a major 
deterrent to understanding. It was considered desirable to avoid a 
wide range of subject matter because of the impossibility of providing 
a cross-section of book properties in a relatively small collection 
unlimited by subject. 
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who responded to the advertisement participated in the experiment. All 
but ten experimentees^ were University students, of whom thirty-four 
were undergraduates and sixty were graduate students. The ten non-students 
included two University employees, six recent graduates, and two high 
school students attending a University-sponsored workshop. 

During the course of the testing, experimentees were not informed 
of the objectives of the experiment. The designers of the experiment 
reasoned that if experimentees began thinking in terms of catalog infor- 
mation, this frame of reference would dictate the types of information 
they vould remember. As a further control for bias, all instructions 
were standardized and mimeographed; a minimum of verbal direction was 
given by the examiner. The results obtained appear to confirm the assump- 
tion that experimentees were indeed ignorant of the fact that their 
recall of specific items of information was being investigated: had 

they desired to perform well, they could easily enough have memorized 
details such as authors and titles during the exposure session. 

A total of approximately three hours' time was required of each 
experimentee: one hour for the exposure session, and two hours for the 

testing session. No time limit for the testing session was set; indi- 
viduals worked at their own pace and varied in the exact amount of 
time they required to complete the session. Normally only one experi- 
mentee was being tested at any one time, and he was alone in the room 
with the examiner. Overlaps occasionally occurred when an experimentee 
took an unusually long amount of time and the next scheduled experimentee 
appeared for his appointment. This caused little inconvenience, and it 
is not believed that any distraction which may have ensued influenced 
the exper imentee ' s per f ormanc e . 



^Hereafter the term "experimentee" will be used to refer to those 
people who participated in the book-memory experiment; the term "subject" 
will be reserved for designating the description of books' content, 
i.e., topic, in the library sense. 
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Exposure Session 

At the first of the two sessions, the exposure session, each 

1 

experimentee was presented with a mimeographed list of thirty books 
and asked to place a check mark in front of the ten which appeared to 
be of greatest interest to him. He was then told to go to the book 
stack housing the experimental collection, locate the ten books, and take 
them back to his desk. After a cursory examination of the ten books, he 
was to assign eAoh a rating. A, B, C, or D, indicating the strength of 
his interest in the book, and mark that rating on the list. He next 
chose five of the ten books, examined them more closely, and composed 
for each a brief comment expressing his reaction. The resultant five 
comments and the books they represented became the focus of the second 
or testing session of the experiment. 

Experimentees were required to supply written comments on the 
five books crucial to the experiment in order to provide them later with 
some kind of purpose for attempting to lemember descriptive character* 
is tics of the books examined. That is, if an experimentee remembered 
having seen a book with a certain title, or of a certain color or size, 
etc. , but remembered nothing about the content of the book or why it 
interested him, such a recollection would be meaningless--the experimentee 
would have no motive to remember anything about the book. Thus the 
comment-“the experimentee ' s own statement of what he found interesting 
about a particular book--was later used to provide him with a reason 
for wanting to describe that book. 

Testing Session 

Two weeks after the exposure session, each experimentee returned 
for' a testing session. This session consisted of two parts; (1) Testing 



^Six different lists of thirty books each were used, thus every 
book in the experimental collection was offered to some experimentees. 

A sample copy of one of the lists, together with copies of all other- 
materials used in the book -memory experiment, is included in Appendix A. 
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of recall of inf oration about the five books perused and commented on, 
and (2) Testing of recognition of all ten books examined, both by author 
and title, and as physical objects. 

Recall testing was done in three successive stages designed to 
elicit first spontaneous, then stimulated recall. This was accomplished 
by first asking the experimentee to volunteer information, and then in 
the second and third stages providing him with cues to memory. 

First, the experimentee was given five blank 5" x 8" cards and 
asked to reconstruct as accurately as possible each of the five comments 
he had written and to give as much additional information about each book 
as he could recall. When he indicated that his memory of the books was 
exhausted, the second stage was initiated. He was then given five check- 
lists (see Appendix A) which inquired about a number of characteristics 
that he might or might not have already recalled, or that he might not have 
considered important. Keeping the cards he had previously written but 
using a different color pen (in order to provide a visual record of how 
frequently and during which stages impressions were recorded or altered), 
he then filled in the checklists with whatever details he was able to 
supply. He was asked not to change any statements he had made on the 
cards but write his then current impressions on the checklists, which 
he numbered to correspond with numbers on the matching cards. At the 
third stage, the experimentee was given a typed copy of the five comments 
he had written during the exposure session (edited to remove specific ref- 
erences to author or title) and asked again to supplement the information 
he had already given if he could do so. He wrote any additional or cor- 
rected information on the same checklists used in the previous stage, but 
used a third color pen. If, on seeing his comments, he recalled books 
which he had not remembered during stages one and two, he was given addi- 
tional checklists and he then proceeded to provide information on those books. 

At this stage he was also asked to compose a title that would de- 
scribe the contents of the book and to indicate subject headings he would 
consult in order to find the book in a library catalog. Unfortunately, 
this refinement was not added to the original design until after about 
one-half of the experimentees had completed their testing session. It 
was initiated because the research staff noted a tendency on the part of 
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experimentees to be quite unspecific in their description of the subject 
of a book, thus making any assessment of the quality of subject informal:. 
tion elicited extremely tenuous. Thus, there is a distinction between 
subject descriptions (informal, narrative statements about the contents 
of the books) and subject headings (estimations of formal catalog entries) 
supplied in the experiment. 

Finally, the experimentee was asked to match the comments that had 
been returned to him to the written information he had provided for each 
of the books 'by numbering the comments to correspond to the cards and the 

checklists. 

Recognition testing was also carried out in three stages: 

(1) author-title recognition, (2) recognition of the actual books in a 
different environment, and (3) recognition of the books in their original 
environment. These stages simulated actual retrieval in that they tested 
the experimentees' ability to recognize "known" books on a list and in 
a bookstack among other books. 

At the first stage, the experimentee was given a mimeographed 
list of some 600 books, listed by author and title. The books actually 
used in the experiment were included in the list, together with several 
"false leads," and the experimentee was required to identify all ten of 
the books he had examined, distinguishing those five he had commented 
on from the other five. At the second and third stages of recognition 
testing, the experimentee was instructed to locate and remove the ten 
books from a bookstack, first from an auxiliary stack in which they 
were shelved at random, and in which he was required to identify the 
books on the basis of their external appearance alone, and finally from 
the position in the stack in which they had been located at the time of 
his first experience with them two weeks previously. At this final 
stage, the experimentee was allowed to examine the textual material in 
the books if he wished in order to determine that they were indeed the 
correct books. At both the second and third recognition stages, he 
indicated which of the books were those he> had commented on and matched 
the books to the written information he had provided for each. 



If I 
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Classification of Data 

One hundred four experimentees were tested, representing a 
potential yield of 520 sets of book-memory data (104 experimentees x 5 
books). As was expected, however, not every experimentee was able 
recall all the books he had commented on; some could remember only t^o 
or three books; some remembered and produced data on books that they 
examined but not commented on, either mistakenly or expecting that 
substitution would be acceptable. Thus, a total of 531 sets were col- 
lected. A criterion for acceptable data was formulated, and all data 
sets that failed to meet the criterion were discarded. The criterion 
was based on the comnent requirement: (1) Data sets must have been 

produced in reference to books for which comments had been written, and 
(2) During the testing session, the data sets must have been correctly 
matched to the appropriate comments. The significance of the comment 
criterion was in ensuring tnat the experimentee actually had a specific 
book in mind and could remember it clearly enough to recognise the state- 
ment he had previously made about it. Thus, the experimental situation 
approached a library situation in which a patron wishes to locate a 
specific book with which he has had previous contact and now wishes to 
retrieve. Ultimately, 440 eligible data sets were identified. 

These 440 data sets, each set representing one experimentee ' s 
s^mory of one book, were further classified in regard to (1) whether or 
not the experimentee was able to substantially reproduce his comment at 
the first stage of recall testing,^ (2) the stage of recall testing 
during which the data set was begun, and (3) the experimentee 's success 
or failure in the two types of tasks during recognition testing. 

Eligible data sets comprise five classes (Classes 1-5) with two 
types of failures^ (b, c failures) possible within all but Class 1. 



^This is not the same as the criterion of demonstrated 
nition of the as 

one reineinbered book. 

^An "a" failure was also defined: The experimentee wm unable 

to match tl typfrc^oJ^e^t to the correct data set. Such failures did 

not meet the criterion for eligible data. 
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Clasees and failures are defined as follows; 



Classes 

1 Data set begun at Stage 1 of recall testing; 

experiinentee reproduced comment fairly 
accurately; no occurrence of any failures. 

2 Same standard as Class 1, but one or more 

failures occurred. Thus Class 2 contains 
all responses which met Class 1 criteria 
except for containing b and/or c failures. 

3 Data set begun at Stage 1 of recall testing; 

comment was not reproduced; b and/or c 
failures possible. 

4 Data set begun at Stage 2 of recall testing 

with checklist; b and/or c failures possible. 

5 Data set begun at Stage 3 of recall testing 

with return of typed comments; b and/or c 
failures possible. 



Failures 

b Experiinentee was unable to recognize the object 

book in the original books tack during Stage 3 
of recognition testing. 

c Experimentee was unable to recognize the object 

book on the mimeographed list during Stage 1 
of recognition testing. 

be The two above failures in combination. 

The 440 data sets were also divided into two groups representing 
field of academic specialization--psychology, and other disciplines. 

This categorization was imposed in order to simulate a comparison of 
the performances of specialists and laymen in the field. Results of 
this comparison are discussed in “Evaluation of Variations in Response 
Patterns of Psychology and Non-Psychology Students for All Character- 
istics," which follows the present paper. 

Table 1 shows the distribution of the 440 eligible data sets 
into classes and failures. It can be seen that over half (243) of all 
eligible data sets fell into Class 1, the class that presumably contains 
those responses based on the strongest recall of particular books. It 
was informally hypothesized that responses in these data sets would 
evidence the most accurate memory of book properties; however, although 
memory in some cases was slightly more accurate, it was not greatly so. 



o 
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TABLE 1 

CLASSIFICATION OF ELIGIBLE DATA SETS 

(N = 440) 



Class 


No 

Failure 


Type of Failure 


Total 


b 


be 


c 


1 


243 


* 


* 


* 


243 


2 


* 


2 


8 


53 


63 


3 


42 


4 


4 


17 


67 


4 


10 


0 


0 


0 


10 


5 


39 


1 


6 


11 


57 


Total 


334 


7 


18 


81 


440 



Cells containing asterisks are empty by 
definition; other empty cells are those in which 
no data sets happened to fall. 



Analysis of data sets yielded some general observations concern- 
ing the nature of experimentees ’ memory and their conceptual approaches 
to book characteristics. First, responses elicited during the first 
stage of recall testing (writing information on blank cards) were usually 
related to intellectual aspects of books, i.e., subject content and 
personal reactions, titles and authors; physical and bibliographic fea- 
tures other than title and author were infrequently mentioned. However, 
if bibliographic or physical features were mentioned at this stage, 
descriptions were accurate in most cases. Second, it can be seen that 
the ma j ority - -near ly 85 per cent--of books recalled were recalled at 
Stage 1 (Classes 1, 2, 3). Stage 2 (Class 4), presentation of the 
checklist, stimulated the recall of only ten books, and Stage 3 (Class 5), 
return of copies of the original comments, resulted in recall of fifty- 
seven additional books. Finally, information about bibliographic and 
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physical characteristics was elicited chiefly during Stage 2 when exper- 
imentees filled out checklists/ It had been planned to measure the 
effect of the memory stimuli provided by the checklist and the typed 
comments, but information provided at Stage 1 was minimal, as were 
changes made at Stage 3 (with the exception of those fifty-seven re- 
spondents who remembered object books only when they saw their own 
comments on those books— Class 5 responses). Therefore, in the final 
evaluation of data, the stage of recall testing during which information 
W3.S elicitsd wa.s not taksn into considoration. 
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‘since the greatest portion of data analyzed was provided by 
experimentees at the second stage of the recall portion of the book- 
memory experiment, a discussion of the nature and peculiarities of the 
checklist employed is pertinent. The checklist (questionnaire) was given 
to experimentees after they had freely recalled as much infonration as 
possible about the books they had examined previously. Thus it was in- 
tended to serve as a stimulus to memory by suggesting categories of 
information which the experimentees might have overlooked and by pro- 
viding a structured framework of choices within some categories. Several 
types of questions were utilized, each type having been considered the 
most appropriate for eliciting information in the category suggested. 
Although desiring to obtain as "natural" information as possible, the 
researchers believed it necessary to provide a^structure for items on 
which scattered responses due to experimentees lack of familiarity with 
possible descriptions was expected. This may have been a mistaken 
assumption, since experimentees sometimes seemed unable to select a 
single choice or to comprehend distinctions between choices. The three 
types of items appearing on the checklist may be described as ( ) s ort 
answer or open-ended response, (2) multiple choice, and p) binary or 
ternary choice (roughly equivalent to true-false items with an uncertain 
option). The effect of this variation in the form of questions on 
quantity and quality of responses elicited has not been determined; 
however, one observes that the multiple choice items functioned essen- 
tially as forced choice items, while the short answer and ternary choice 
items elicited a higher percentage of null responses or don t ^nw 
answers. The possibility of indiscriminate guessing cannot be ruled 
out, especially in the case of items for which multiple options were 
supplied. Another factor which may have influenced the data is the 
well-known tendency of students, who comprised the major portion of the 
group of experimentees, to avoid leaving any question unanswered. 
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Findings of the Book-Memory Experiment 

Results Relative to Retrieval of Known Works 

From the classification of data (p. 10 above) and from a check 
of experimentees ' responses against entries in the catalog of the Uni- 
versity Library,^ a measure of library users' performance in various 
areas strategic to retrieval of previously consulted works may be 
obtained. Table 2 summarizes these performance ca4>abilities. 



« TABLE 2 



DEMONSTRATED CAPABILITIES OF BOOK-MEMORY EXPERIMENTEES 

(N = 440) 





Description of Capability 


Number 


Percentage 




Experimentees capable of providing a 
complete author-and-title citation 


10 


2.3 




Experimentees capable of providing 
author or title data sufficient to 
locate the object book in catalog 


110 


25.0 


> 


Experimentees capable of providing 
author or title or subject data 
sufficient to locate the object 
book in catalog 


312 


70.9 




Experimentees capable of recognizing 
the object book on a list of 
approximately 600 books 


341 


77.6 




Experimentees capable of recognizing 
the obj.ect book either on list or 
in books tack, or both 


422 


95.9 



These measures of retrieval-oriented capabilities yield several 
striking observations. First, recall of the most critical catalog entries, 
author and title, is relatively infrequent. The number of complete 



^See "Effectiveness of Book-memory Data for Conventional Catalog 
Retrieval," which follows. 
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citations recalled is extremely small; if the requirements for author- 
title retrieval are extended to admit cases in which either author or 
title was recalled accurately enough to permit location of the book in 
the catalog, still only one-fourth of respondents would be successful in 
retrieval attempts. However, if author or title or subject information 
is utilized, the retrieval rate increases to 71 per cent. 

Comparison of items one and four in Table 2 indicates that although 
free recall of accurate citations is seldom demonstrated, the ability to 
recognize accurate citations is extremely good. As was Stated previous Xy> 
the list in which the citations appeared was long and formidable, con- 
sisting of a total of approximately 600 citations. Yet, confronted with 
this massive listing, 78 per cent of respondents were able to select an 
accurate citation to an object book. 

Finally, it is seen that recognition of the object book either 
on a list or in a bookstack is far more accurate than is recall of 
critical catalog entries. Ninety-six per cent of respondents recognized 
actual object books or their bibliographical representation. Thus, it 
seems reasonable to conclude that individuals who cannot recall sufficient 
information to permit immediate retrieval of works with which they have 
had previous contact would be able to recognize the relevant work among 
a fairly large number of works which are possibly relevant. 

Accuracy of Recall of Book Characteristics 

A primary objective of the book-memory experiment was to investi- 
gate the "memorability" of certain common characteristics of books. It 
is of interest to know what characteristics people most frequently think 



^The relationship between the third and fourth items in Table 2 
has not been explored fully; a case could be made for reducing the 
figures for the third item by a factor of 78 per cent (the percentage 
of respondents capable of recognizing a printed record of author and 
title), reasoning that this figure represents the percentage of respon- 
dents who would indeed recognize a catalog card describing the o J^^t 
book. If this reduction were effected, a figure of 57 per cent would 
result for the third item above. 
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they remember, and if memory of these characteristics tends to be accurate 
enough to aid in describing books for retrieval in library catalogs. 

Table 3 presents a comparative summary of responses to checklist 
items regarding various characteristics of books. The responses tabu- 
lated represent experimentees ' final judgment on each item irrespective 
of the stage at which it was elicited. It will be noted that some items 
included on the checklist have not been tabulated with this summary; in 
most cases this is due to the paucity of responses to the item. 

Book characteristics presented in Table 3 are arranged in order 
of the proportion of substantive responses given by experimentees. The 
first column of figures, "Percentage Substantive Responses," indicates 
the proportion of respondents supplying either a correct or incorrect 
response, as opposed to those who either left the item blank or indicated 
that they didn't know or didn't remember the nature of the characteristic. 
Thus, a substantive response, whether it be correct or incorrect, indi- 
cates that an experimentee thought he remembered the ch^^rac tens tic 

in question. 

Column two, "Percentage Substantive Responses Correct," provides 
an evaluation of the accuracy of experimentees' memory: When a person 

believed he remembered a characteristic (as evidenced by his making a 
substantive response), did he remember it correctly? The percentages 
in this column, then, are derived from the numbers represented in the 

first column. 

The third column, "Percentage Correct of Total Response," indi- 
cates the percentage of correct responses within the total set of correct 
incorrect, and null ("don't know" responses and items left blank) 
rssponsBS to 0£ich characteristic* 

The last column, "Item on Checklist," gives the number of the 
item or group of items under which the characteristic appeared on the 

^See Appendix B for explanation of those items not tabulated 
and analyzed. 
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TABLE 3 



memory of book characteristics 
(N = 440) 



Characteristic 



Percentage Percentage 
Percentage substantive Correct 
Substantive j^gg^onses of Total 
Responses 



Responses 

Correct 



Subject description . . 

Level of readership . . . 

Type of work ...»•• 

Work in a single volume . 
Subject headings .... 
Condition ....«• ^ ■ 

Number of pages (50-page 

intervals) 

Type of binding .... 
Chapters titled . 

Problems included at ends of 

chapters 

Height (1/2" intervals) . . . 

Title 

Tables included 

Case studies included . . . • 

Translation from another 

language 

Figures, charts, diagrams 

included 

Date of publication (decade) 
Graphs included . 
Illustrations included 
Index included 
Footnotes included .... 

Preface 

Color of binding 

Glossary included 

Bibliography included . . . 

Quotations from literary 

works included 

Light and dark color cover 
Reprint or revision .... 
Position in bookstack . . . 

Dedication 

Detail of cover design . . 

Place (city) of publication 

Author 

Position on shelf 

Type of publisher 

Published in series .... 
Publisher’s name . . . . , 

Call number (BF only) . . . 



99.5 

99.1 
93.8 

97.7 

97.4 

95.6 

91.6 

88.7 

88.4 

84.8 

83.4 

82.7 

74.5 

74.1 

73.8 

73.2 

73.2 

72.0 

69.8 

68.0 

65.7 

64.3 

63.2 

61.8 
56. 8 

55.7 

55.4 

53.4 

37.5 

29.3 

28.4 
23.0 

20.9 
20.2 
18.2 

13.9 
7.3 
5.2 



76.0 

53.8 

44.1 

97.4 

73.3 

33.6 

23.8 

71.5 

91.8 

95.8 

24.2 

27.4 

79.6 

73.4 

88.6 

83.0 

41.6 

84.2 

81.8 

64.2 

63.6 

87.3 
29.8 

89.7 

56.8 

72.7 

63.9 

83.3 

55.8 

61.3 

52.8 

28.7 

76. 1 

47.2 
50.0 
83.6 

46.8 
73.9 



Responses 

75.6 

53.4 

43.6 

95.2 
70.8 

32.2 

21.8 

63.5 
81.1 

81.4 

20.2 

22.7 

59.3 

54.3 

65.4 

60.7 

30.4 
60.7 

57.1 

43.6 

41.1 

56.1 

18.9 

55.4 

32.3 

40.5 

35.4 

45.2 

20.9 

17.9 
15.0 

6.6 

15.9 

9.5 
9.1 

11.6 

3.5 
3.9 



Item on 
Check- 
list 

5 
4 
3 
9 

Added 

6 

6 
6 
9 

9 
6 
2 
9 
9 

8 

9 
8 
9 
6 
9 
9 
9 
6 
9 
9 
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checklist used in Stage 2 of recall testing. Complete summaries of dis- 
tributions of responses to individual characteristics are presented in 
Appendix B to the present report. The Appendix is arranged in order of 
items' appearance on the checklist. 

The distinction between the percentages appearing in the second 
and third columns is an important one. The proportion of correct to non- 
correct responses regarding some characteristics shifts drastically as 
the base is changed from all respondents (N = 440) to just those respon- 
dents who made an assertion about the characteristic (N = 440 - null 
responses). See, for example, the esponses for Author and for Call 
number --relatively few of the total respondents correctly described 
these characteristics, but considering only those who attempted a descrip 
tion, the tendency was to be correct. On the other hand, characteristics 
which the majority of respondents attempted to describe show little 
difference between the two measures of correctness. 

For convenience of discussion, book characteristics have been 
grouped into four categories: (1) the intellectual aspects of books-- 

authorship, content, nature of the work, (2) the facts of publication, 

(3) the physical description of the book, and (4) all binary character- 
istics --those features which are not common to all books but which may 
or may not be present in a specific book. 

Memory of Intellectual Aspects of Books 

Table 4 presents a summary of findings regarding memory of in- 
tellectual characteristics: author, title, subject description, subject 

headings, type of work, and level of readership. The percentage of 
respondents providing correct, incorrect, and null responses to each 
characteristic are shown. The last column in the table, "Response Choices 
on Checklist," indicates whether the item as it appeared on the checklist 
required experimentees to formulate their own responses or if they were 
given a choice of responses. 

Except for subject headings, which are applied to a book in a 
library environment, each of these characteristics is an integral and 



-18- 



TABLE 4 

MEMORY OF INTELLECTUAL CHARACTERISTICS OF BOOKS 

(N=440) 



Charlie ter is tic 


Distribution of Responses 
(in percentages) 


Response 

Choices 

on 




Correct 


Incorrect 


Null 


Checklist 


Author 


15.9 


5. o' 


79.1 


no 


Title 


22.7 


60.0 


17.3 


no 


Subject description 


75.7 


23.8 


.5 


no 


Subject headings 
(N=192) 


70.8 


26.6 


2.6 


no 


Type of work 


43.6 


55.2 


1.2 


yes 


Level of readership 


53.4 


45.7 


0.9 


yes 



unchanging aspect of a particular work. They generally are important 
to the book user--whether he realizes it or not--both for understanding 
and for identification of the book. Moreover, this group contains the 
characteristics which comprise the three most common approaches to 
traditional catalog retrieval: author, title, and subject. Data from 

the book -memory experiment affords an indication of the relative ten- 
dencies to remember these specifics. It should be noted once more that 
questions regarding author, title, and subject information were answered 
from unaided memory--no clues or possible choices were given respondents. 

Author . --A key retrieval element is the author if a desired work, 
whether personal or corporate. All books in the experimental collection 
were considered to have personal authors or editors. The checklist item 
requesting the name of the author of the object book was given prominence, ' 

being the first item on the checklist. Responses to the author item were 
tabulated within descriptive categories, and occurred as shown in Table 5. 
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TABLE 5 

RECALL OF AUTHOR'S NAME 



Type of Response 


Number 


Percentage 


Correct first and last name 


37 


8.5 


Correct last name, first initial 


2 


.5 


Correct last name only 


31 


7.0 


Incorrect name 


22 


5.0 


(Misspelled or partial name 
only, 10 cases; wrong author 
named, 12 cases) 






No response 


348 


79.0 


Total 


440 


100.0 



Sixteen per cent of the responses may be considered correct (the 
first three items in the table) in that they provide at least minimal 
identification of the author. Only 5 per cent of responses are incorrect, 
while the item was left blank in 79 per cent of cases. This suggests 
that respondents may be disinclined to guess at an author's name; although 
the frequency of correct responses is disappointingly low, incorrect 
responses account for fewer than one in four of total responses. The 
high proportion of null responses (79 per cent) suggests that in the case 
of vaguely remembered works, attempted retrieval by author may be futile. 

Supplementary information about authors provided by respondents 
was largely of a nebulous character, e.g., "two authors" (no name pro- 
vided), "a woman," "a psychologist." Several responses, however, connected 
the author with an institution, e.g., "a psychologist at Duke University," 
"formerly at University of Chicago," etc. This type of information 
might prove useful, at least in local cataloging. As might be expected, 
the majority of this type of response identified University of Chicago 
authors. A third type of response connected the author with .another of • 
his works, e.g., "author of Client-Centered Therapy ." 
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Title . --A second key retrieval element tested was recall of the 
title of the object book. It is interesting to note that in. comparing 
title responses to author responses, almost a complete reversal of correct 
incorrect, and null responses is observed. Title responses were analyzed 
and categorized; findings are shown in Table 6. 

The checklist item dealing with title provided respondents an 
opportunity to suggest alternative possibilities if they believed their 
first-named title might be incorrect. If, in this "second try," the 
correct title was given, it was tabulated as a correct title in Table 6; 
in such cases, the first-named title was disregarded. 

TABLE 6 



recall of title 



Type of Response 


Number 


Percentage 


Correct 




100 




22.8 


Incorrect 

Synonym error 
Permutation 
Incomplete 
Unrecognizable 

Total incorrect 

Null 


84 

30 

55 

95 


264 

76 


19.2 

6.8 

12.5 

21.5 


60.0 

17.2 


Total 




440 




100.0 



Alternative title possibilities provided in the responses indicate 
people's awareness of synonym confusion. Analysis of incorrectly stated 
titles shows that synonym errors comprise the largest single group of 
recognizable errors (19.2 per cent of all cases), and occurred even more 
frequently than null responses (17.2 per cent of all cases). 

In comparing title responses to author responses, it is seen 
that while respondents appear reluctant to guess at an author s name, 
no such inhibition governs statements of titles. Although only 21 per 
cent of respondents provided a name purporting to be that of the author 



o 



- 21 - 



of a given work, the likelihood is that that name is correct. By con- 
trast, nearly 83 per cent of respondents supplied a title; however, the 
likelihood is that the title as stated is inaccurate. 

An exploratory analysis of title responses was undertaken by a 
member of the research staff in an effort to identify types of errors 
which could be exploited for retrieval purposes. A number of interesting 
observations emerged from this study, a detailed report of which appears 
in "An Analysis of Non-Exact Title Data," attached to the present report. 

Subject . --The third common retrieval element is a subject heading 
assigned to a book. Two types of subject data were collected in the 
book -memory experiment. The original plan was to extract subject speci- 
fications from experimentees' descriptions provided during the three 
stages of recall testing. Responses elicited in this manner were often 
difficult to interpret and seemed to bear little resemblance to actual 
headings, so an additional requirement was initiated: Experimentees 

were instructed to list subject headings under which they thought the 
object book might appear in the University Library catalog (see instruc- 
tion for Part II, sheet 3a, Appendix A). The two entries in Table 3 
(page 16 above), ref lect this distinction: the first, "Subject descrip- 

tion," indicates data gleaned from respondents' free verbalizations 
(N = 440), and the second, "Subject headings," indicates data provided 
in response to the added instruction (N = 192). The nature of subject 
information given by respondents has been the focus of a separate investi- 
gation, reported in "Accuracy and Utility of Subject -Re la ted Responses," 
attached to the present report. Investigators checked subject responses 
(both descriptions and headings) against the actual headings applied to 
object books and the seventh edition of the Library of Congress Subject 
Heading list in order to determine whether or not a respondent's subject 
specification would ultimately, through following cross references if 
necessary, lead to the object book. Findings are summarized in Table 7, 
which shows the percentages of success in providing immediately and 
ultimately correct subject headings. Slightly over 18 per cent of 
respondents who provided formal subject headings would find the object 
book directly under the heading exactly as given, without being required 
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to consult variant forms or subdivisions of that heading or being directed 
to more appropriate headings j 71 per cent of subject headings provided 
would lead, through cross-references, to the exact heading. Contrary 
to the investigators’ early pessimistic appraisal, nearly 30 per cent of 
subject descriptions in the form of phrases or sentences contained words 
or phrases which fortuitously proved to be the exact headings assigned 
to object books; 76 per cent would ultimately lead to the desired books. 

TABIiE 7 



congruence; of subject .responses •- 

TO ACTUAL SUBJECT HEADINGS 



Type of 
Response 


Accuracy of Response 


Base N 


Directly 

Congruent 


Ultimately 

Congruent 


Total 


Unstructured 

subject 

description 


29.8 


45.8 


75.6 


440 


Estimated 

subject 

heading 


18.2 


52.6 


70.8 


192 



A check of author, title and subject information provided by 
respondents against the University Library catalog showed that 46 per 
cent of respondents had provided a subject description adequate to locate 
the object book either directly or through cross references. Although 
undoubtedly many instances of catalog failure and searcher failure con- 
tributed to the discrepancy between the two ’’success rates” (46 per cent 
vs. 76 per cent), the discrepancy does suggest that methodical, exhaus- 
tive searching of a type more likely to be effected by a machine than 
by a human searcher bears promise of fruitful results. 

Type of work . -“In order to describe books in terms of content 
and intent, eleven categories of works (plus an "other" category) were 
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identified within the experimental book collection. Examples of types 
are handbook, dissertation, textbook, history. Experimentees were asked 
to indicate the category which most closely described the object book. 

They were allowed to check more than one category if they thought it 
necessary; 29 per cent of respondents did so. Including both single and 
multiple responses, 43 per cent of respondents correctly categorized 
object books, 56 per cent did so incorrectly, and only 1 per cent failed 
to respond to the item. 

Experimentees apparently found this item difficult. In addition 
to the 138 responses where two or more categories were checked as appro- 
priate, 46 cases (9.6 per cent) occurred where the respondent checked 
the category "other," often giving a description of what he considered 
the book to be. The project staff's interpretation of these phenomena 
is that respondents were confused by terminology and by the large number 
of categories from which to choose, and that confusion might have been 
reduced by providing fewer and broader categories together with defini- 
tions of what each category comprised. 

Level of readership . --The level of readership of a book was 
defined as being the audience (or purpose) for which it was intended. 

All books in the experimental collection were assigned to one of four 
levels: (1) popularization, (2) treatment for the generally educated 

layman, (3) work for beginning college students, (4) work for advanced 
students or professionals. These four categories plus an "other" option 
were listed on the checklist in a multiple choice arrangement; experi- 
mentees were allowed to check more than one response if they thought it 
necessary. Approximately 24 per cent of responses were multiple 
responses — more than one option had been checked. A multiple response 
was considered correct if one of the options checked was the correct 
one; if neither was correct, the response was considered wrong. The 
tendency on the part of respondents was to assign books to a higher 
level than the researchers had done. This may be attributable to the 
fact that the researchers were attempting to use the general population 
as a referent when assigning levels; perhaps the experimentees were 
thinking in terms of a university population. At any rate, only slhirghlrly^ 
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more than half (53.4 per cent) of the respondents were able to recall 
the approximate level of object books although 99 per cent attempted an 
answer. The element of subjective judgment enters in here; for this 
reason, any notation of the level of a work on its catalog representation 
would necessarily be open to question. 

Memory of Facts of Publication 

A second group of characteristics includes facts about the 
publication of a book. Table 8 summarizes findings relative to these 
characteristics. It is interesting to note that three of the four 
characteristics are normally noted in cataloging practice and are 
considered indispensable bibliographic information, but none seems to 
be greatly memorable. 

t 

TABLE 8 

MEMORY OF FACTS OF PUBLICATION 
(N=440) 



Characteristic 


Distribution of Responses 
(in percentages) 


Response 

Choices 

nn 


Correct 


Incorrect 


Null 


LL 

Checklist 


Publisher 


3.5 


3.8 


92.7 


no 


Type of publisher 


9.1 


9.1 


81.8 


no 


City of publication 


6.6 


16.4 


77.0 


no 


Date (decade) 


30.4 


42.8 


26.8 


no 



Publisher . -“Slightly more than 7 per cent of the 440 respondents 
attempted to supply the name of the publisher of the object book, and 
these responses were nearly evenly divided between correct and incorrect 
specifications. Only 3.5 per cent of all respondents were able to correctly 
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name the publisher. With the exception of Penguin Books, which was 
correctly identified three times, only two publishers were correctly 
identified on more than one occasion: University of Chicago Press and 

Free Press, both of which are well-known to University of Chicago 
students. Seventy-four publishers were represented in the experimental 
collection; those correctly identified and the number of correct identi- 
fications for each are as follows: University of Chicago Press (2), 

Duke University Press (1), Free Press (2), Grove Press (1), Iowa Uni- 
versity Press (1), Naturegraph (1), Pantheon (1), Penguin Books (3), 
Philosophical Library (1), Prentice-Hall (1), Yale University Press (1). 

While it is generally agreed that the identity of a publisher 
is an important item in describing a particular book, it appears that 
few people take notice of or remember publishers if they have no reason 
to do so. 

Type of publisher . --Responses describing the type of publisher 
of object books were equally divided between correct and incorrect-- 9 per 
cent of the 440 respondents provided correct descriptions. On the check- 
list the question regarding type of publisher was of the open-ended 
variety; no choice of responses was suggested. Thus, reponses were for 
the most part verbalizations which had to be interpreted. Six types of 
publishers had been identified within the experimental book collection 
(trade, university, educational, government, society, author). Responses 
such as "general" were interpreted as meaning "trade" and were judged 
correct; an example of a response judged incorrect was "not scholarly" 
in description of a university press. Responses to this item illustrate 
the terminological difficulties encountered when choices (or explanations 
of terms) were not offered. It may be concluded that lay persons are 
not aware of the distinctions among types of publishers and their char- 
acteristic output. 

Place of publication . --Twenty- three per cent of respondents 
gave information purporting to be the place of publication of an object 
book; about one-third of these responses were accurate. Only the name 
of the city of publication was considered correct. Most of the incorrect 
responses were names of countries, e.g.. United States, England. Predictably 
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the city most frequently identified correctly was New York. Also, 
when a respondent had correctly identified a publisher as a specific 
university press, the city of publication was usually known; however, 
since this instance arose only four times, the generalization may be 

suspect. 

nste of publication .— Thirty per cent of respondents were able 
to specify within the decade (1930's, 1950's, etc.) the date of publi- 
cation of an object book. Nearly as many (27 per cent) failed to respond 
to the item. Responses to the item were equivalent to free verbaliza- 
tions, •some respondents naming a single year, and others giving varying;- - 
spans of years. Where spans of years (e.g., 1963-65, 1920’s) were given, 
the midpoint in the span was considered to be the response. 

When considering a characteristic that is continuously distributed 
as is date, the problem in evaluating responses and also in planning 
retrieval strategy is one of determining the latitude of error that can 
be tolerated. A small latitude, as two or three years, may be quite 
specific in identifying a manageable segment of a collection of books, 
however, few persons seem able to specify dates very closely. A wider 
latitude, as ten or twenty years, will admit more correct responses, but 
is correspondingly less specific. For example, 264 (60 per cent) of the 
440 object books were published since 1950, a span of less than two 
decades. Thus a date specification with a broad span, as 1960's would 
identify a very large portion of the collection. 

Mpmorv of Physica l Characteristics 

The third group of book characteristics includes those charac- 
teristics which are associated with a book's physical description or 
location. Some of these characteristics may be considered permanent, 
as number of pages or height; others may change, or a variation between 
copies of the same work may exist. Thus, should any of these non-constant 
characteristics be considered for catalog notation, steps would have to 
be taken to ensure the updating of the catalog or the permanence of the 

characteristic. 



Table 9 presents a summary of responses pertaining to books' 
physical characteristics. As in previous tables, the right column, 
"Response Choices on Checklist," indicates whether or not the item was 
in multiple-choice form on the checklist. A "no" in the column means 
that respondents structured answers themselves- -no suggestions were given. 

Hei ght . --Books in the experimental collection ranged in height 
from seven to eleven inches, with the majority clustered about the median 
of eight and one-half inches. Respondent's specifications of heights 
of object books ranged from five to fourteen inches. During recall 
testing, a set of "sample books" labeled with their actual heights was 
on display for experimentees to consult. The sample books were pro- 
vided after the pilot study showed an amazing incongruity between actual 
book heights and respondents' specifications. The researchers concluded 
that experimentees needed help of some type in judging heights since they 
apparently had little conception of size. Even with samples clearly 
visible, respondents' memory of book height is not impressive. Although 
83 per cent of respondents attempted to state the height of object books, 
only 20 psr cent were able to specify actual heights within one-half inch. 

Height is presently noted in catalog descriptions of books, 
but findings of this experiment suggest that it is not particularly 
memorable. Moreover, specifications of height approach meaninglessness 
when referring to books of "average" size, since this is such a large 
group, but when height specifications refer to books of extremely large 
or small size they may be more meaningful. For these reasons, height 
does not appear to be a particularly promising retrieval characteristic. 

Number of pages . --Responses regarding number of pages followed 
a pattern similar to that of responses to height, in that the great 
majority of respondents attempted an answer, but that answer was usually 
incorrect. Twenty-two per cent of respondents were able to specify 
within fifty-page intervals (e.g. , 50-99pp. , 100-l49pp.) the correct 



^Eighty-one per cent of object tn^sOis were between eight and 
nine and three-eighths inches tall. 
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TABLE 9 



MEMORY OF PHYSICAL CHARACTERISTICS 




umber of pages ; 70 per cent were incorrect in their specification. No 
uggested responses or options appeared on the checklist, thus responses 
:ook a variety of forms. When a span of pages was given, the midpoint 
n the span was considered to be the response, which was tabulated within 
■he appropriate fifty-page interval. Although books in the experimental 
:ollection ranged from approximately twenty-five to over 550 pages, half 
If the object books were between 200 and 399 pages in length; 55 per cent 
of correct answers were in reference to these books. 
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Tvpe of binding . --Books in the experimental collection had seven 
different types of bindings: library bindings, publisher's cloth bind- 

ings, library paperback bindings, paper-board covers, pamphlet bindings, 
and leather bindings. These categories appeared the checklist in the 
form of a multiple choice question. Respondents were fairly successful 
in specifying the type of binding of object books : 64 per cent were 

correct, while 25 per cent were incorrect. It must be noted that 75 per 
cent of the object books had cloth publisher's bindings, and books in 
this category accounted for 246 (88 per cent) of the 279 correct responses. 

In light of the fact that large numbers of object books exhibited 
average or typical conditions of the four preceding characteristics (date 
of publication, height, number of pages, type of binding), one should 
consider the distribution of responses which might be expected under 
conditions of random guessing constrained by some knowledge of which 
choices are in general the most likely. The observed distribution of 
responses to object books appears to be similar to an expected random 
distribution, thus suggesting that knowledge of probable distributions of 
book characteristics may be operable to a greater extent than memory 
of characteristics of a particular book. The important implications 
of this point can be more clearly seen from the treatment by W. S. Cooper in 
"The Potential Usefulness of Catalog Access Points Other Than Author, 

Title, and Subject." 

Color . --The checklist item inquiring, into color was a multiple 
choice item (ten colors listed); respondents were requested to check the 
appropriate color to describe object books. Only 19 per cent of respon- 
dents were able to recall color correctly. Nearly twice that many, 

37 per cent, failed to respond to the item, and 44 per cent provided 
an incorrect description. 

Recall (in fact, identification) of color is complicated by two 
factors: (1) the difficulty in determining and classifying the color of 

certain books, and (2) the occurrence of multi-color books. Colors of 
books, unfortunately, are not confined to clean, bright primary and 
secondary colors: the effects of wear tend to obscure the original color 

of the binding, and some colors in their various hues and shades are 
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inherently difficult to distinguish. Second, many books are bound with 
more than one color, for example, some flamboyant multicolored patterns 
arc found, and half -rbindings which have one color on the spine and another 
on the covers are fairly common. 

Prior to the conduct of the experiment, the research group had 
by consensus agreed upon the color of each book in the experimental 
collection. When responses were tallied, each response was judged 
correct or incorrect according to the researchers' prior decisions. 
Multicolored books were considered to be a separate color class, so 
that a total of eleven color specifications was possible. Table 10 
gives an indication of the memorability of the various colors. There are 
no data concerning white books, since none of the object books was white. 
Results pertaining to multicolored books are probably misleading, since 
experimentees received no special instructions regarding multiple colors, 
and many checked just one color. ("Correct" multicolor responses were 
those in which the respondent indicated that the book actually was 
more than one color.) 

The research staff had expected that color would prove to be a 
memorable characteristic, since nearly every book user probably believes 
that he remembers the colors of his books quite accurately. Therefore 
it was surprising to find that color proved to be one of the least 
memorable of all characteristics. Upon reflection, the researchers 
concluded that memory of color of specific books is probably a function 
of frequent use of those books, and that a single casual use of a book 
does not provide motivation to remember the color. 

Light or dark color . --The checklist item regarding lightness or 
darkness of color proved to be an ambiguous one. It was the intent of 
the researchers that respondents consider the item only if they were 
unable to recall the exact color of an object book; however, many respon- 
dents who did recall color also answered this item. Therefore, responses 
to this item provide inconclusive results. It will be noted that even 
though the item was variously construed, nearly twice as many respondents 
(35.4 per cent) accurately described object books as light or dark than 
accurately identified specific colors (18.9 per cent), as might be 
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expected in view of the diminished number of choice categories avail- 
able for responding to each item. 



TABLE 10 

MEMORABILITY OF INDIVIDUAL COLORS 



Color 


Distribution of Responses 
(in percentages) 


Base N 




Correct 


Incorrect 


Null 


Blue 


14.4 


30.0 


35.6 


90 


Brown 


31.6 


42.1 


23.6 


19 


Orange 


0.0 


80.0 


20.0 


5 


Gray 


23.8 


28.6 


47.6 


21 


Black 


24.0 


26.7 


49.3 


71 


Red 


22.2 


43.2 


34.6 


104 


Green 


11.4 


42.8 


45.8 


35 


Purple 


28.6 


57.1 


14.3 


7 


Yellow 


41.2 


35.3 


23.5 


17 


Multi-colored 


8.5 


60.5 


31.0 


71 


Total 




440 



np^ails of cover design . -The checklist item dealing with details 
of cover design was an optional item-lt was intended to have been ignored 
if it didn’t apply to an object book. Sixty-six respondents chose not 
to ignore the item, but wrote in "nothing." These respondents plus 
fifty-nine who checked one of the options provided a total of 125 re- 
sponses. Of these 125 substantive responses, 53 per cent were correct; 
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47 per cent incorrect. Failure to respond to this item cannot be 

interpreted in the same way it is in reference to other items, in that 
a null response may signify either of two conditions: the object book 

didn't have any type of design, etc., on the cover, or, the respondent 

simply didn't recall. 

Condition . -“The condition of object books was correctly recalled 
by 32 per cent of respondents; 64 per cent were incorrect. Five possi- 
bilities for describing condition were suggested: new, fairly new, 

used, worn, and bad. Books in the experimental collection had been 
judged and described by the research staff. Respondents described object 
books in the aggregate as being in better condition than the researchers 
had thought. Indeed, only one object book was described as being "bad" 
by a respondent, while the researchers had identified sixty-six as "bad." 

Describing condition involves a subjective judgment, and the 
actual conditon of a book is, of course, subject to change. These facts 
plus relatively unimpressive memorability, suggest that considering 
condition as a cataloging and retrieval possibility would be unprofitable. 

Call number . --Respondents were asked whether they recalled call 
numbers (Library of Congress classification numbers) of object books. 

There was no instance of a respondent recalling a completely accurate 
call number, but seventeen respondents (4 per cent) managed to recall 
the initial letters of the call number--BF. Of these seventeen, seven 
41 per cent) were given by respondents who were majors or graduate stu- 
dents in psychology, people who might be expected to know classification 
numbers in their field. Thus, it is difficult to say whether call number 
recall is a function of memory or of familiarity with typical classifi- 
cation in the field. It will be noted that nearly 95 per cent of 
respondents made no attempt to state even a partial call number, and 
those who did tended to be correct. 

Position in bookstack and position on shelf . --The books tack in 
which the experimental collection was housed was approximately three 
feet wide and contained six shelves of books. Respondents were asked 
on which shelf the object book had been located and whether it had been 
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on the left, middle, or right part of that shelf. Twenty-one per cent 
of respondents remembered that the object book had been in the top, 
middle, or bottom segments of the bookstackj only 10 per cent remem- 
bered that it had been in the left, middle, or right shelf position. 

There was a high incidence of null responses to both items, indicating 
that physical location is apparently not a memorable characteristic. 

Memory of Binary Characteristics 

The final group of book characteristics to be considered con- 
sists of characteristics which are either present or absent in any giyen 
book, and which have been designated "binary." 

The binary characteristics displayed in Table 11 are arranged 
in order of percentages of correct responses. The column headed ' Per- 
centage of Object Books Possessing Characteristic" shows the relative 
frequency of each characteristic's occurrence; for example, 97.3 per 
cent of the 440 object books were published as a single volume. Columns 
headed "Correct" and "Incorrect" do not distinguish qualitatively among 
responses beyond defining their accuracy. Thus, responses of "yes" 
given that the object book did possess a characteristic, and "no' given 
that it did not, are subsumed under the group of correct responses. 

Null responses are cases in which respondents checked the option don t 
know" or left the item blank. 

Fourteen of the eighteen characteristics considered as binary 
appeared on the checklist as a group of yes-no items (checklist question 
number 9). Two columns of blanks, headed "Yes" and "No" were printed 
on the checklist. In order to discourage indiscriminate guessing, 
experimentees were instructed to supply a third column of blanks headed 

"Don't Know." 

The four remaining items. Translation, Illustrations, Reprint- 
Revision, and Series, did not appear with the above group of fourteen 
but were ultimately tallied in the same manner as the fourteen original 
binary characteristics. 
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TABLE 11 

MEMORY OF BINARY BOOK CHARACTERISTICS 

(N=440) 



Characteristic 


Distribution of Responses 
(in percentages) 


Percentage of 
Object Books 
Possessing 
Characteristic 




Correct 


Incorrect 


Null 


Single volume 


95.2 


2.5 


2.3 


97.3 


Problems 


81.4 


3.4 


15.2 


1.4 


Chapter titles 


81.1 


7.3 


11.6 


95.7 


Translation 


65.4 


8.4 


26.2 


13.6 


Graphs 


60.7 


11.3 


28.0 


20.0 


Figures, charts 
diagrams 


60.7 


12.5 


26.8 


30.9 


Tables 


59.3 


15.2 


25.5 


28.8 


Illustrations 


57.1 


12.7 


30.2 


31.2 


Preface 


56.1 


8.2 


35.7 


91.6 


Glossary 


55.4 


6.4 


38.2 


4.8 


Case studies 


54.3 


19.8 


25.9 


37.7 


Reprint -revision 


45.2 


9.1 


45.7 


19.1 


Index 


43.6 


24.4 


32.0 


68.4 


Footnotes 


41.1 


24.6 


34.3 


69.7 


Quotations 


40.5 


15.2 


44.3 


32.2 


Bibliography 


32.3 


24.5 


43.2 


49. 6 


Dedication 


17.9 


11.4 


70.7 


44.6 


Series 


11.6 


2.3 


86.1 


20.0 
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Translation . --The checklist item regarding translation was composed 
of two parts, the first of which asked merely whether the book was or 
was not translated into English from another* language. Responses to 
this part were treated as were those to other questions regarding binary 
characteristics, and it is seen that 65 per cent of respondents were 
accurate in their responses. The majority of correct responses were 
"no," since only 13.6 per cent of object books actually had been trans- 
lated. The second part of the item requested respondents to identify 
the original language of translated books. Responses here were sparse; 
only twenty-one were elicited, of which twenty were correct. German 
was the language most frequently correctly identified. 

Series . --Experimentees were requested to supply the names of 
series in which object books had been published; three were able to do 
so correctly. A number of respondents offered instead a response of 
"none" or "not a series." These responses were tallied as if the question 
had referred to a binary characteristic; thus findings are possibly not 
indicative of respondents’ memory patterns, since the intent of the 
item was sacrificed in order to deal with the data that was gathered. 

Illustrations . --Experimentees were asked to describe the kind 
of illustrations (if any) which appeared in object books. A number of 
respondents supplied descriptions of types of illustrations, e.g., 
photographs, drawings; others merely answered yes or no. It was decided 
to treat all these responses as binary, that is, in terms of yes or no. 
Thus, 52 per cent of respondents correctly specified whether object 
books were or were not illustrated. 

Renrint -revision . --The question asking whether or not an object 
book was a reprinted or a revised edition was actually binary in intent 
if not in form. Answers elicited were usually of the yes-no variety; 
occasionally respondents specified that the work was reprinted or that 
it was a revised edition. All responses were treated as yes, no, or don't 

know responses in tallying. 

Simple binary characteristics . "-The remainder of the binary 
characteris tics--those which required no qualitative description by 
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experimentees- -require no special explanation beyond that which was given 
above. It will be noted that the two characteristics eliciting the 
highest proportion of correct responses. Single Volume and Problems, 
appear to be memorable but are only specific in isolating a distinctive 
portion of a book collection in reference to the atypical state. That 
is, 97.3 per cent of object books were single volumes (they were not one 
volume of a set), so a response of "yes" has little specificity; by 
contrast, a correct response of "no" has high specificity. The reverse 
is true of problems--a book that does have problems is in the minority 
case, and a positive statement that a book does in fact have problems 
denotes high specificity. 

Several of these binary characteristics are now noted in typical 
catalog representation, others are non-standard catalog data. Undoubtedly 
each of them would have significance to some catalog user at some time, 
but their significance as possible retrieval points is dependent upon 
their memorability and their distribution throughout a collection. From 
memory data alone, characteristics with low memorability can probably 
be eliminated from consideration as retrieval points. 

Summary and Discussion 

Findings of the book-memory experiment reported here provide 
information of potential value in planning future catalogs. First, an 
indication of the memorability of various book characteristics is of 
value for determining which characteristics book users note and which 
ones they ignore, at least in a casual, weakly-motivated exposure situa- 
tion. Characteristics that appear to be memorable are worthy of further 
investigation into their potential utility in retrieval systems. Next, 
one must take into account the "statistical usefulness" of these clues. 

That is, even a memorable characteristic, such as a broad subject 
classification, may be of small value if many books are similarly 
classified. It is possible to estimate the probable increase in re- 
trieval efficiency that would be gained by provision of additional catalog 
access points to books, a problem investigated and reported by W. S. Cooper 
in "The Potential Usefulness of Catalog Access Points Other Than Author, 

Title, and Subject." 
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Data gathered in this experiment allows us to compare the 
memorability of three sets of characteristics: those which presently 
comprise standard entries or approaches in alphabetic catalogs --primarily 
author, title, subject headings; those which presently comprise elements 
of catalog records; and those which presently are not described on 
catalog records. 

In present alphabetic catalogs, retrieval of desired works may 
be effected only through consulting one or more of the standard entries 
assigned to each work. Usually, each work has at least three such 
entries, author, title, and one or more subject headings. Findings of 
the book-memory experiment indicate that subject headings are the best 
specified of these access points. 

Recalling the distinction between subject headings and subject 
descriptions, we note that both varieties of subject information were 
found to be potentially effective for retrieval of object books in more 
than 70 per cent of cases. Eighteen per cent of headings and 30 per 
cent of descriptions provided by respondents proved to match exactly 
the actual subject headings applied to object books. It must be kept 
in mind that all experimentees were asked to provide subject descriptions, 
while less than half of them were asked to suggest approximations of 
formal subject headings, resulting in unequal data bases; still, it is 
surprising that unstructured subject descriptions allow as good access 
to object books as estimated subject headings do. Of even greater interest 
is the observation that unstructured descriptions more frequently proved 
to be exactly accurate headings than were the attempts to guess actual 
subject headings. Why this should be the case is a matter for speculation- 
are library users inhibited by the system of formalized subject headings? 
Would a system of subject description other than that of the Library of 
Congress provide improved access and easier use? Would regular updating 
of assigned subject headings to keep them consistent with current termi- 
nology and usage improve access? 

Turning to other standard catalog entries, we see that the author 
and the title of previously -examined books are remembered less frequently 
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and with less accuracy than is subject information. Approximately 
16 per cent of authors were recalled, 23 per cent of titles. There is, 
of course, no method of approach to titles or authors corresponding 
to the syndetic approach to subject headings in present catalogs; thus, 
a library user with an inaccurate author or title specification is at 
a dead end. To further decrease the chances of retrieval by title, 
certain types of titles are commonly not represented by title entries 
in catalogs, for example, "nondistinctive" titles. Analysis of elicited 
title data revealed no consistent patterns of errors capable of exploi- 
tation, and no efficient method of utilizing separate elements of title 
specifications for retrieval. It was observed, however, that there 
appears to be an inverse relationship between correct title specifica- 
tion and the length of the title-respondents demonstrated better recall 
of short titles than of long ones. Also, the data indicates that a 
permuted title index would permit access to many titles presently 
unretrievable. 

Of the standard means of access, then, subject headings (or some 
method of subject description) appear to be the most memorable. Further 
research into library users’ conceptual processes as applicable to 
subject specifications and methods of representing subject specifications 
on catalog records should lead to improved retrievability through subject 

approaches . 

The memorability of descriptive elements presently included in 
catalog records appears to be low. It is observed that publication 
information is infrequently remembered accurately, as are facts about 
a book's physical description. 

Nonstandard catalog information, that is elements not recorded 
under present cataloging rules, evidences a wide range of memorability. 

A number of generalizations may be made regarding memorability of these 
characteristics. First, characteristics relevant to a book's existence 
as a physical object are poorly remembered --in fact, surprisingly so. 
Color is an outstanding example of this phenomenon, as are a book's 
condition and details of the appearance of its cover. An item appeared 
on the checklist inquiring into object books' format; responses to the 
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item were infrequent enough to preclude further consideration. Second, 
characteristics evidencing better memorability were most frequently 
those bearing on aspects of books* intellectual content. Problems, 
chapter titles, illustrative matter such as tables, figures, etc., 
preface, and glossary elicited fairly accurate responses, and all may 
be characterized as intellectual aids to the reader. These character- 
istics also have high visual impact, a fact which may contribute to 
their memorability. Furthermore, in regard to illustrative matter, it 
is interesting to note that cataloging rules have consistently encouraged 
noting of various types of graphic presentations (e.g., charts, photo- 
graphs, plates) when their inclusion in a work seems significant; however, 
the trend is toward the simpler method of describing most non-text 
material as "illus.*' 

Findings of the book-memory experiment indicate tendencies to 
remember certain characteristics of previously -used books; however, to 
conclude that these characteristics are indeed ’’memorable" and therefore 
probably significant for retrieval is to ignore other factors which must 
be considered as well. At least four considerations involved in assessing 
particular characteristics' potential utility for retrieval of specific 
works can be suggested. 

1. Memorabrnti- --The retrieval utility of a given characteristic 
is highly dependent upon accuracy of recall (human memory). Inaccurate 
recall obviously impedes the retrieval process, while absence of recall 
contributes nothing. Purely £or the sake of illustration, let us consider 
the following criterion of memorability: A majority of all respondents 

(51 per cent or more) correctly described the characteristic in question, 
and, discounting respondents who made no attempt to describe it, the 
proportion of correct to incorrect responses is at least two to one. 
Application of this criterion reveals that fourteen characteristics are 
likely to be specified accurately (see Table 3, page 16 above). These 
fourteen are Single volume. Problems, Chapter titles. Glossary, Translation 
Preface, Graphs, Figures-charts-diagraras, Illustrations, Tables, Case 
studies. Subject description. Subject headings. Type of binding. Other 
characteristics exhibit varying degrees of memorability which may or may 
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not be capable of exploitation for retrieval; the fact that they do not 
meet the memorability criterion is, of course, not to imply that they 
may not be valuable elements of a catalog record. 

Any measure of memorability which is derived solely by comparing 
percentages of accurate and inaccurate responses is. of course, suspect. 

Such an oversimplified measure can indicate only observed response pat 
terns. It does not take into account any deviation of responses from 
an expected random distribution, and thus cannot claim to measure actual 
recall The relationship of randomness to memorability has been explored 
by W. S. Cooper and is reported in his paper. "The Potential Usefulness 
of Catalog Access Points Other Than Author, Title, and Subject." 

2. Permanence. - -A second consideration is of a given character- 
istic's permanence and consistency over time, over various copies of 
the same work, and of location. Some characteristics are immediately 
recognized as temporary or varying, for example, condition, type of 
binding-even color may vary between original publishers' bindings and 
rebindings. (These two latter characteristics could, of course, be 
controlled in libraries.) Another group of characteristics may be more 
subtly variant, as in different libraries (or even the same library at 
different times), for example, call number, location in stacks. It can 
be seen, however, that none of these possibly impermanent characteristics, 
with the exception of type of binding, meet the first criterion of memor- 
ability, a fact which is interesting in itself. 

3. iinamhiPiiitv . --Frequently, evaluations of the nature of a 
given characteristic are ambiguous, are 'relative to a given clientele, 
or are the result of various subjective interpretations. In some cases, 
clear definitions are lacking. To cite an example, level of readership 
is clearly relative to a specific clientele; a work considered difficult 
or advanced by the lay public may be elementary to a specialist in the 
field. Another example of an ambiguous characteristic is color. Blue 
is recognizable, so is green, but between blue and green on the spectrum 
lie a number of shades which are difficult to define. Also, library 
terminology may prove a problem to library users, as may subtle distinctions 
which users have no cause to notice. Catalog designers should maintain 
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awareness of the possibility of misinterpretation of descriptive elements 
and either eliminate ambiguous elements or make clear definitions avail- 
able to the library's clientele. 

4. Specificity . --Retrieval utility of a given characteristic 
is also dependent upon its capacity to define a manageable portion of a 
total collection. That is, the presence, absence, or quality of Charac- 
teristic X should define a specific percentage of the collection. Letting 
represent the books possessing a given quality of the characteristic 
and C the total collection, we note that specificity increases as the 
proportion X^/C decreases, and is of negligible value when X^/C is a 
large proportion. Thus, the fact that a given book has an original 
cloth publisher's binding identifies approximately 51 per cent of the 
collection as candidate books; the fact that a book has a pamphlet bind- 
ing identifies about 16 per cent of the collection. Characteristics 
which have only two possible qualities (presence or absence) vary greatly 
in their specificity. This phenomenon should be considered in designing 
retrieval systems, and search strategies developed which will take 
primary advantage of specifications which define fairly small propor- 
tions of a collection. 

The papers which follow report investigations into several 
characteristics' actual and potential retrieval utility. These investi- 
gations were conducted by members of the research group and were based 
on data collected in the book-memory experiment. These reports, together 
with the findings of the book-memory experiment, present various per- 
spectives for viewing cataloging and retrieval practices. The information 
presented and the conclusions reached contain valuable insights into the 
problem of retrieval of specific works. 
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EVALUATION OF VARIATIONS IN RESPONSE PATTERNS OF PSYCHOLOGY AND NON- 
PSYCHOLOGY STUDENTS FOR ALL CHARACTERISTICS 

Helen F. Schmierer 

As noted earlier (cf. "Memorability of Book Characteristics: An 

Experimental Study," p. 4) books used in the experiment were drawn from 
the psychology (BF) collection of the University of Chicago Library. One 
might expect that psychology students would provide more accurate responses 
to questions concerning these books than non- psychology students on such 
characteristics as author, title, subject of call number, if only because 
the psychology student is presumably more familiar with the literature in 
the field. It was therefore thought profitable to evaluate responses in 
terms of experimentee background in order to establish whether psychology 
student responses were significantly different from those of the non- 
psychology student. 

Responses which comprised the data base (440) were separated into 
two groups according to the psychology or non-psychology status of experi- 
mentees. The initial step in comparing performances was accomplished by 
applying the chi-square test; results were evaluated at 95 per cent con- 
fidence level. Responses concerning nine characteristics (publisher, 
condition, call number, problems at end of chapter, type of publisher, 
translation, type of binding, place of publication, and type of work) 
showed statistical significance. 

The data for these nine characteristics were then examined to 
determine the nature of the variation between the performance of the 
psychology majors and the remainder of the respondents . For each of the 



^Experimentees were asked to report their current academic status 
during the exposure session. Psychology student is defined as a psychology 
major (if undergraduate) or as student within the psyf.hology department 
(if graduate student). 

2 

See tables in Appendix C. 
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two classes of experimentees the following percentages were calculated 
for each of the nine characteristics: 

(A) number of correct responses 

X 

(B) Number of correct responses - number of incorrect responses 

X 

(x being total number of responses within a class: psychology, seventy-six; 

non-psychology, 364). (A) gives simply the percentage of correct responses, 

1 

while (B) was designed to take into account the effect of erroneous replies. 

In both computations higher percentages indicate superior perfor- 
mance. Negative (!) percentages from (B) indicate that the number of in- 
correct responses exceeded the correct ones. Performances on both measures 
are shown in Table 1. 

\ 

In three lnstances--problems, call number and condition of book-- 
psychology students performed better consistently. In three other cases-- 
translation, publisher and type of work--non-psychology students con- 
sistently evidenced better performance. In the three remaining cases 
performance was not as simple to evaluate. 

For only one of the characteristics (call number) for which it was 
hypothesized that psychology students would perform better was a statis- 
tically significant difference in performance between psychology and 
non- psychology students found. Psychology students did perform better 
in this case. 



For the purpose of this evaluation null responses were not in- 
cluded. Although it can be hypothesized that a null response is better 
than an incorrect response, it is impossible to assign any real value to 
a null. 
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TABI£ 1 



COMPARISON OF RESPONSE ACCURACY SHOWN BY PSYCHOLOGY 
AND NON-PSYCHOLOGY STUDENTS 
ON NINE STATISTICALLY SIGNIFICANT CHARACTERISTICS 





Percentage 


Correct 




Characteristic 


Measure A 


Measure B 




Non- 

Psych 


Psych 


Non- 

Psych 


Psych 


Type of work 


47.4 

* 


25.0 


- 3.6 


-50.0 


Condition 


29.6 


44.7 


-36.2 


- 7.9 


Type of binding 


64.3 


59.2 


37.6 


40.8 


Problems 


78.9 


93.4 


75.2 


90.7 


Translation 


68.0 


53.9 


59.0 


48.9 


Place 


7.4 


2.6 


-11.2 


- 2.6 


Type of publisher 


10.4 


2.6 


0.0 


0.0 


Publisher 


4.1 


0.0 


-0.3 


- 1.3 


Call number 


2.7 


9.8 


1.3 


7.9 



However, in evaluating the raw data for all characteristics 
showing statistically significant differences in response performance 
no logical pattern of variation between psychology and non- psychology 
students can be seen. Therefore, it is reasonable to conclude that 
experimentee background had no appreciable effect on performance. 




EFFECTIVENESS OF BOOK-MEMORY DATA FOR 
CONVENTIONAL CATALOG RETRIEVAL 

Delores K. Vaughan 

Information provided by respondents in the Book-Memory exper- 
iment was used to search the University of Chicago Library catalog in 
an attempt to assess the retrieval effectiveness of information recalled. 
For each book recalled by each experimentee, a search memo was prepared, 
recording data concerning the authbr, title, and subject description or 
subject headings provided by the experimentee. 

Author and title information was transcribed exactly as given by 
respondents, i.e., their statement of the author's name, the title of 
the work, and any alternative possibilities suggested for either category. 
If subject heading possibilities were given, these were transcribed with 
subject data. The bulk of the subject data transcribed was in the form 
of key words or phrases extracted from the description of the subject of 
each book provided by the respondent. 

In cases where respondents provided different or contradictory 
information during the succession of experimental stages, the informa- 
tion provided at the latest stage was used, assuming that this represented 
the respondent's final judgment. Where no information was given, this 
part of the memo was left blank. 

In this manner, one search memo was prepared for each book (object 
book) recalled by each experimentee. Each search memo specified (1) the 
actual author and title of the object book, and (2) the information des- 
ignating author} title, and subject as provided by the experimentee. 

The search was conducted at the public catalog of Harper Library 
(the main catalog of the University Library collections). All of the 
books used in the Book-Memory experiment were known to be represented 
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(at least, minimally) in this catalog. Each memo was searched individ- 
ually by as many of the three approaches (author, title, subject) as 
possible. Individual searches terminated when all possible approaches 

had been attempted. 

progress and results of the searches were recorded on the memos: 
the existence or non-existence in the catalog of a suggested entry or 
heading, the total number of cards entered under each heading consulted, 
and, when a book was found, the position of the appropriate card among 

those bearing the heading, e.g., the 45th of 70 cards. 

Each memo was searched through all approaches provided— if the 
experimentee had provided author and title and subject information, all 
three were utilized; theoretically the object book could have been "found" 
by all three approaches. In cases of respondents' providing several 
alternative possibilities under one approach, the searcher utilized only 
as many alternatives as were necessary to find the object book.. For 
example, if three subject headings had been provided and the object book 
was located through the first one searched, the remaining two headings 
were not searched. If only one type of information was provided, then, 
of course, that approach was the only one attempted. Searches were 
considered successful when a catalog card representing the object book 
was located, failures when all possible entries and headings had been 

searched unsuccessfully. 

Four categories of approaches have been identified for summarizing 
catalog search results: (1) complete citation, that is, author's complete 

and correct name plus the title of the work, (2) author only, in varying 
degrees of completeness or correctness, (3) title only, and (4) subject 
description and/or suggested subject headings only. Results are shown 
in Table 1 in terms of the disposition of each search memo: the object 

book was either found or not found. In the cases of books' being found 
under multiple approaches, a schedule of priority of successes was 
asslgned-a complete citation take s priority over author, author over 
title, title over subject-so that each book was considered "found" under 

only one approach. 
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Of the 440 memos searched, 312 (71 per cent) of the object books 
were successfully located through at least one approach; 128 (29 per cent) 
were not located. Of the 312 successful searches, fifty-two were found 
through more than one approach, usually by subject in addition to author 
and/or title. However, in Table 1 only the first (priority) find for 
each memo is tabulated. Table 1 shows the distribution of approaches 

under which object books were found. 



TABLE 1 

RESULTS OF CATALOG SEARCHES ON INFORMATION 
GIVEN BY RESPONDENTS 



Result of Search 


Number 


Percentage 


Object book found by 
complete citation 


10 


2.3 


Object book found through 
author entry 


58 


.13.2 


Object book found through 
title entry 


52 


11.8 


Object book found through 
subject heading 


192 


43.6 


Object book not found 


128 


29.0 


Total 


440 


100.0 



The length of individual searched varied from one to more than 
2,000 catalog cards. Larger numbers of cards were consulted in subject 
approaches than in author of title approaches. The number of cards 
bearing a given subject heading (including subdivisions) ranged from 
one to 1,550. Most subjects specified by respondents were of a general 
or broad nature and consequently represented large numbers of cards, 
usually between 300 and 600. It is Important to note that the data 
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which follow represent the nuttO^ers of cards consulted under various entries 
or headings prior to and including the location of the relevant card; they 
do not show the total number of cards possibly relevant. For example, 
there might be seventy- five cards filed under a given author entry; if the 
relevant card was the twentieth one, that search length is twenty cards 
(plus the number of cards searched under other entries, if any, prior to 
location of the relevant entry). Data on actual search lengths is pre- 
sented in Tables 2 and 3 which follow. 

Table 2 indicates the length of individual successful searches in 
spans of numbers of cards. Only one successful find per search memo is 
shown; figures are for priority finds. Table 3 indicates search lengths 
for all finds““if one search memo yielded two or more successes under 
different approaches, all are tabulated. The approach category "complete 
citation" has been distributed into its separate component approaches; 
the ten successes in this category were added to both the author and 

the title rows. 

Comparing Table 2 with Table 3, we see that the greatest number 
of additional successes came from subject approaches: 234 search memos 

contained subject information that led to the object card; 192 of these 
subject successes had been priority successes, thus forty-two respondents 
had provided accurate (or adequate) subject information in addition to 
accurate author or title information. 

Again comparing Tables 2 and 3, we note no great difference in 
the distribution of search lengths as they are grouped. Although it is 
seen that subject searches involve the manipulation of the greatest numbers 
of cards, the addition of the non-priority subject successes did not 
increase the average search lengths. 

The distribution of search lengths according to approach category 
shows quite clearly that a complete citation (author's full name plus the 
title of the object book) provides the most expedient approach to the 
catalog; however, few respondents were able to provide a complete citation. 
The next most expedient approach is title--no title searches involved 
consulting more than ten cards. Title entries are, of course fairly 



o 



r 






-49- 



TABLE 2 

LENGTH OF SEARCHES (PRIORITY SUCCESSES) 



Approach 


Number of Cards Manipulated 
in Locating Object Book 


Total 


Direct 


2-10 


11-50 


51-100 


100+ 


Complete 














citation 


10 


• • 


• • 


• • 


• • 


10 


Author 


• • 


24 


20 


14 


• 

• • 


58 


Title 


46 


6 


• • 


• • 


• • 


52 


Subject 


T • 


28 


38 


39 


87 


192 


Total 


56 


58 


58 


53 


87 


312 


Percent 


17.9 


18.6 


18.6 


17.0 


29.9 


100.0 



TABLE 3 

LENGTH OF SEARCHES (MULTIPLE SUCCESSES) 



Approach 


Number of Cards Manipulated 
in Locating Object Book 


Total 


Direct 


2-10 


11-50 


51-100 


100+ 


Author 


• • 


34 


20 


14 


• • 


68 


Title 


66 


10 


• • 


• • 


• • 


76 


Subject 


• • 


33 


47 


50 


104 


234 


Total 


66 


77 


67 


64 


104 


378 


Percent 


17.5 


20.4 


17.7 


16.9 


27.5 


100.0 



ERIC 
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distinctive; it is unusual to find more than two books with the same 
title, although the cataloging of multiple editions of the same book 
can generate several title cards. 

The success of using an author approach is dependent upon several 
factors: how much of the author's name the patron has--last name only 

or last and first name, how many authors' names are possible candidates 
(happily, no respondents named "Smith" as an author), and how prolific 
the object author is. The probability of locating an object book 
through an author entry appears to decrease with increasing search length; 
no author entries were located beyond 100 cards. 

By contrast, it is seen that subject successes occur at a fairly 
consistent rate irrespective of the number of cards manipulated. Table 4 
illustrates this phenomenon with data relative to total successful 
searches (Table 3). Success in locating subject entries may be strongly 
dependent upon perseverance- -the patron must be willing to keep trying 
until he locates the appropriate heading either through his own knowledge 
or through following a chain of cross-references, often a discouraging 
process . 



TABLE 4 

DISTRIBUTION OF SUCCESSFUL SUBJECT SEARCHES 
BY NUMBER OF CARDS MANIPULATED 



Number 
of Cards 


Number of 
Successes 


Percentage 
Succ jsses 


2-10 


33 


14.1 


11-50 


47 


20.1 


51-100 


50 


21.4 


100+ 


104 


44.4 



Total 



234 



100.0 
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Figure 1 portrays graphically the rate of success in locating 
object catalog cards as a function of the nunJ^er of cards manipulated. 

All approaches-author. title, and subject, are included. The categorxes 
Direct and 2-10 have been combined (refer to Tables 2 and 3), the 
category 100b has been arbitrarily defined as averaging 300 cards, prob- 
ably a conservative estimate, since many searches involved manipulatxng 
1 000 cards or more. It is seen that the success rate decreases sharply 
al the number of cards increases; thus if a library patron has fairly 
accurate information, usually author or title, about the particular 
work for which he is searching, he usually does r-^.t have to consult an 
inordinate number of cards. However, if information is incomplete or 
non-specific, as in the case of a significant amount of subject informatxon, 

a longer search is likely to be required. 



FIGURE 1 

location of object cards as a function 

OF SEARCH LENGTH 




Number of cards manipulated 
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It has been pointed out that the shorter searches (fewer than 100 cards) 
are usually for author or title entries; the longer ones (over 100 cards) 
are for subject entries only. 

It is interesting as well as informative to compare the quality 
of responses regarding author, title, and subject information elicited in 
the book-memory experiment to the results obtained in the catalog search. 
One would expect that the two "success rates" be similar. Data presented 
in Table 5 permits comparison of these two performances. 

The immediately apparent discrepancies indicate that catalog 
pQj^formance falls short of its potential. Failure to locate in a catalog 
an entry which had been accurately stated may be due to a number of 
causes: catalog failures, as misfiling, absence of a critical entry; 

searcher failures; and common cataloging practices. 

TABLE 5 

CATALOG USEFULNESS OF RESPONSES ELICITED 
IN BOOK-MEMORY EXPERIMENT 



Approach 


Response 

Judged 

Correct 


Catalog 

Entry 

Located 


4 


Author 


70 


68 




Title 


100 


76 




Subject 


333 


234 





Two standard cataloging practices contributed to the majority of 
failures to locate title entries. First, if the title of a book is the 
same as a subject heading applied to it, for example, Psychology > or 
Hypnotism , a title card is not provided--the subject entry is considered 
to take priority. In the catalog search, the location of such a subject 
card was not counted as a successful title search. Second, title cards 
are normally not provided for non-distinctive titles, as Introduction to 
Psychology ; thus, such title searches failed. In addition, a number of 
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title cards were never located because, 



inexplicably, they simply had 



not been provided. 

Searches in pursuit of subject entries were the most frustrating 
of all. First, in spite of libraries' attempts to keep the number of 
entries at any given catalog access point to a minimum, there are single 
subject headings extending over as many as 600 cards; when subdivisions 
of a single subject heading are included, often as many as 2,500 cards 
are Indicated. For example, the heading CHILD STUDY, subdivided into 
three sections by date of publication, covers 1,550 cards. Form and 
content subdivisions, e.g., PSYCHOLOGY-ADDEESSES , ESSAYS, LECTURES; 
PSYCHOLOGY-COLLECTIONS, and PSYCHOLOGY, HISTORY (to name only a few) 
force the patron to familiarize himself with the system and to formulate 
his subject specification quite accurately. Information given on search 
memos frequently was not adequate to locate even an appropriate subject 
heading, much less the object entry. Second, the catalog's syndetic 
structure of cross references provides patrons assistance in getting from 
an incorrect to a correct subject heading. In the case of "see" refer- 
ences, this aid is valuable; in the case of "see also" references, the 
patron, confronted with a list of possibly twenty alternative headings 
to consult, may give up in despair. 



The group of 128 search memos that yielded no catalog search 
successes under any approach was analyzed for the cause of failures. 

The most common cause of author-or-title-approach failure was the complete 
absence of any information; if incorrect author or title information was 
given, no method of manipulating that information in order to make it 
operative in catalog searching is available. Subject information, on the 
other hand, has a greater possibility of being functional, since theoret- 
ically, at least, a vague subject specification may, through the cross- 
reference system, lead ultimately to the object entry. Since in only two 
of the 128 cases no subject description was given, 126 attempts to 
locate subject entries were fruitless. Analysis showed that the most 
common causes of subject-search failure were (1) heading specified was 
too broad or general (forty cases) , (2) heading specified was grossly 
inaccurate (thirty-nine cases) , (3) critical entries were'misplaced 



-54- 



in the catalog (twenty-three cases), and (4) searcher failed 
to locate anexisting catalog card (eighteen cases). Of these four 
leading causes of subject failures, the first two reflect human cognitive 
errors; the latter two, human manipulative errors. Failures, due to this 
latter type of error presumably could be completely eliminated in a 
mechanized retrieval system, as could time-consuming and frequently in- 
effective lengthy subject searches through a maze of cross-references. 

Summary 

Using information provided by respondents in the Book-Memory 
experiment, a catalog search was conducted. Nearly 71 per cent of 
respondents had provided author, title, or subject information that was 
accurate enough to locate object books in the card catalog. Subject 
heading appro£(ches provided the greatest number of successful searches— 
over half of the object books could be located through this approach. 

Less than 2.0 per cent of object books could be located through title 
approaches; 15 per cent through author approaches. 

The catalog search yielded several noteworthy observations. 

First, the rate of success over all approaches declines as the number 
of cards manipulated increases. Author and title searches, however, 
account for the majority of successes within a small number of cards; 
subject searches involve much larger numbers of cards. Second, except 
for a complete and accurate citation, title approaches afford the most 
efficient access to specific books, since titles tend to be unique 
specifications. Finally, certain cataloging practices and human errors 
account for a significant number of search failures, thus card catalogs 
are prevented from functioning at their potential level of effectiveness. 



ACCUKACY AHD UTILITY OT SUBJECT-BELATED RESPONSES 

Dorothy A. Day 

In the book-memory experiment, experimentees were given three 
opportunities^ to provide information pertinent to subject headings: 
at Stage I (card) they were asked to give anything they could remember 
about the book; at Stage II (checklist question 5) they were asked to 
describe the subject of the book "as specifically as you can in a few 
words." Approximately half-way through the experiment, when the data 
had been accumulated in sufficiently large amounts to determine some 
problems, it appeared that these two sources provided highly unstructured, 
rambling responses from which phrases would have to be extracted-- 
hence introducing an element of subjectivity on the part of the person 
extracting these phrases. Therefore the remaining experimentees 
(accounting for 192 of the AAO responses) were requested, after com- 
pleting the questionnaire, to tell what subject heading they would 
consult in Harper catalog^ to find the book (stage II(3a) or SH question). 
The responses to this question were on the whole brief and required no 
further manipulation to be used for purposes of analysis. Since no 
restraint was put on the number of headings given, a single response 
could provide as many as ten headings; the average number was between 
two and three headings per response. 

For those responses presented in a rambling form, key words or 
phrases were extracted and listed for each response. These extracted 
phrases, plus the responses which were usable in the form given, were 

then compared with: 

(1) the subject headings actually used for each book as traced 
on the card in Harper catalog, and 

’•See questionnaire and instruction sheets in Appendix A. 

^The comprehensive public catalog of the University of Chicago 
Libraries, located in Harper Library. 
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(2) the cross references (see and see also) given to each of 
these headings in the seventh edition of the Ltbyaty 
Congress list of subject headings. 

Matches or near matches were tallied in the following categories. 

A. COMPLETE HEADING GIVEN 

OK: heading given matched perfectly with correct heading 

or cross-reference 

2. FORM: heading given was a form variation of the correct 

heading or cross-reference 

a. P/S: plural- singular variation (e.g., allergy vs. 

allergies) 

b. AB: abbreviated form given by respondent (e.g., correct 

form psychology /respondent gave psych.) 

c SP: spelling error in form given by respondent - 

(e.g., correct form fairies /respondent gave faeries) 

d. OTHER FORM: some other form variation (e.g., correct 

form hypnotism/respondent gave hypnosis) 

3. INV: inverted form given by respondent 

form child study /respondent gave study of child) 

B. PARTIAL HEADING GIVEN (first part of heading correct or nearly so) 

NS: no subheading given by respondent (e.g., correct form 

psychology-history/respondent gave psychology; correct 
form thought and thinking/respondent gave thought) 

2. WS: subheading given by respondent was wrong (e.g., correct 

form psychology— history/respondent gave psychology- 
religion) 

For both of these categories (NS and WS) : 

a. MOK: main part of heading correct 

b. MP/S: plural-singular variation of main part 

of heading 

c. MAB: abbreviated form of main part of heading given 

d. MSP: spelling error in main part of heading 

e. M FORM: some other form variation in main part 

of heading 

These headings were called "usable" for finding the book. Our 440 respond- 
ents provided 491 such headings. The remaining headings given (those not 



falling into one of the above categories) were not analyzed. 

In contrast to other book features tested in the experiment, 
results on subject headings were extren^ly good. Of the 440 respondents 
only two failed to provide some sort t.f subject information; of the 
remaining 438 respondents, 333 provided information judged "usable" 
(falling in the above categories) for finding the book by subject 
approach— over 75 per cent of the respondents. 



TABLE 1 



RESPONDENTS GIVING USABLE HEADINGS 
(All data) 



No usable headings 

At least one usable heading 

At least one direct 
At least one cross-reference 
At least one of each 

Total 



Null 

Total 



105 



171 

93 

69 

333 

2 

A40 



In respect to this finding, some remarks should be made. First, 
classification of a subject as "usable" sometimes assumes unlimited 
patience on the part of the patron in dealing with the catalog, for 
it includes anything which can be traced, directly or through cross 
references, to the actual subject heading used, and assumes ploughing 
through all cards under that heading until the exact card is hit upon. 
This can sometimes involve upwards of 600 cards under one heading alone 
and up to twenty cross references to be checked. 

Separate data was also kept for the persons asked to give a 
specific subject heading for the book (SH question): 




^nnlv a careful estimate can be made of the total nu^er of 
headings acLally given by experimentees, because "headings often ha 
to be extracted from phrases and frequently involved the staff s 
udgLn^ as to what qLlified as a heading, ^he figure arrived .at by . 
this process is 1,173 headings for our 440 respondents. 
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TABLE 2 

respondents giving usable headings 

(SH question) 

No usable headings 

At least one usable heading 

At least one direct 57 

At least one cross-reference 56 

At least one of each ^ 

Total 

Null 
Total 

Of the 192 such responses, five failed to provide any heading, and 136 
provided at least one usable heading (71 per cent of those asked). While 
differing somewhat in nature from the data extracted from the other 
two sources, this data givfes a figure quite close to the first (76 per cent) 

The figures above are for responses the experimentees gave for 
each book, representing whether the responses provided at least one 
usable' heading in each case. Many of our experimentees gave several 
responses, frequently providing more than one usable heading for each 

book: 



136 

5 

192 



TABLE 3 

RESPONDENTS GIVING USABLE HEADINGS 

(All data) 



No usable headings 

At least one usable heading 

One 210 

Two 95 

Three 2A 

Four 2 

Five i. 



Total 

Null 

Total 



105 



333 

2 

AAO 



o 



TAB.JJE 4 

respondents giving usable headings 

(SH question) 



No usable headings 

At least one usable heading 
One 87 

Two 45 

Three 
Total 

Null 

Total 



51 



136 

5 

192 



Overall, 123 of the respondents provided two or more usable headings 
(28 per cent of the 440). On the subject heading question, forty-nine 
of the 192 gave two or more usable headings (25.5 per cent). 

Of the persons giving ^ least one usable heading, 123 of 333 
gave more than one usable heading (37 per cent); for the SH question, 
forty-nine of 136 such respondents gave multiple usable headings (36 
per cent). This would seem to indicate that at least when specifically 
asked for a subject heading, there is some tendency to give multiple 
usable points of access to the book. A retrieval system which took 
advantage of such multiple points of access would hence greatly increase 

the findability of the book. 

Of the 491 usable headings given by our 440 respondents, there is 
a rather strong tendency to exact matches : 



TABLE 5 

USABLE COMPLETE HEADINGS' 



TABLE 6 

USABLE X^ARTIAL HEADINGS' 



Exact 

match 


Form 
var . 


INV 


Total 


MOK 


M Form 
var. 


Total 


Direct 151 


25 


7 


183 


Direct 79 


22 


101 


Cross- 115 
ref. 


16 


3 


134 


Cross- 62 
ref. 


11 


73 


Total 266 

f 


41 ‘ 


10 


317 


Total 141 


33 


174 


1 

For explanation 


of abbreviations. 


see p. 2. 
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Of 317 usable complete headings, direct or cross-reference, 266 were exact 
matches (84 per cent) . This may be partially accounted for by the fact 
that many headings are one word. Of 174 usable partial headings, 141 
have the main part of the heading correct (81 per cent). Among the 
variations allowed as "usable", none significantly distorts the heading 
in findability; all are close enough to the correct heading once in the 
catalog. The data is partly distorted by the categories imposed; if the 
heading given in the response were very far off, it would not be usable 
in the present catalog. However, it is surprising to see so many exact 
matches from our 440 respondents. 

There remains the extremely interesting study of the content of 
the subject headings given by the exp6rimentees . Quite often the 
experimentee gave a heading which, in the judgment of the staff, 
accurately described the subject of the book, yet was not used as .a 
heading for that book in Harper catalog. Under the present catalog, this 
information is useless. Careful study of the headings given might 
suggest possible improvements for the present catalog’s subject heading 
system and also directions any other retrieval system might take. 

Another possible area of exploration would be the study of the 
relation between the subject of a book and its title, particularly 
where the title is an "informative” one. Preliminary study, reported 
elsewhere in this set of papers (see "Title, Subtitle and Table of Contents 
as Sources of Index Words"), seems to indicate that even when an 
experimentee could not give a correct title when asked, his subject 
response usually contained key words from the actual title, subtitle, 

or chapter headings. 

Supplementary data detailing the nature of subject headings pro- 
vided by experimentees is presented in the appendix following. 
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APPENDIX^ 

TABLE A 

USABLE RESPONSES BY TYPE OF HEADINf 



Complete Headings 





OK 


P/S 


AB 


SP 


OTHER 

FORM 


INV 


Total 


Direct 


151 


11 


3 


2 


9 


7 


183 


Cross- 

ref. 


115 


6 


• • 


• • 


10 


o 


134 


Total 


266 


17 


3 


2 


19 


10 


317 






Partial 


Headings 












No Subheading 










MOK 


MP/S 


MAB 


MSP 


M FORM 


Total 




Direct 


50 


5 


1 


• • 


7 


63 




Cross- 

ref. 


28 


• • 


• • 


• • 


2 


30 




Total 


78 


5 


1 


• • 


9 


90 








Wrong Subheading 










MOK , 


MP/S 


MAB 


MSP 


M FOl^M 


Total 




Direct 


29 


5 


• • 


1 


3 


38 




Cross- 

ref. 


34 


1 


1 


9 9 


7 


43 




Total 


63 


6 


1 


1 


10 


81 





1 



For explanation of abbreviations in headings, 



see 



page 2. 
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TABLE B 



USABLE RESPONSES BY CLASS OF RESPONDENT 
Complete Headings 



Direct 



Cross-reference 





nif VAR Total 




OK VAR Total 




Categ. 

1 


101 


16 


117 


Categ. 

1 


38 


12 


70 




Categ. 

2-5 


50 


16 


66 


Categ . 
2-5 


57 


7 


64 




Psych. 

people 


(26) 


(3) 


(29) 


Psych . 
people 


(20) 


(2) 


(22) 




Total 


151 


32 


183 


Total 


115 


19 


134 










Partial Headings 


















No Subheading 














Direct 






Cross- 


•reference 






MOK 


MVAR 


Total 




MOK 


MVAR 


Total 


Categ. 

1 


23 


2 


25 


Categ. 

1 


16 


• • 




16 


Categ. 

2-5 


27 


11 


38 


Categ . 
2-5 


13 


3 




16 


Psych. 

people 


(13) 


W 


(17) 


Psych. 

people 


(2) 


m m 




(2) 


Total 


50 


13 


63 




29 


3 




32 








Wrong Subheading 














Direct 






Cross 


-reference 






MOK 


MVAR 


Total 




MOK 


MV^ 


Total 


Categ. 

1 


20 


6 


26 


Categ . 
1 


21 


6 




27 


Categ. 

2-5 


9 


3 


12 


Categ. 

2-5 


13 


3 




16 


Psych. 

people 


(4) 


(1) 


(5) 


Psych. 

people 


(3) 


(3) 




(6) 


Total 


29 


9 


38 


Total 


34 


9 




43 



^For explanation of categories, see page 10 of main report and 
“Evaluation of Variations of Response Patterns of Psychology and Non- 
Psychology Students for all Characteristics." 
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TABLE C 

USABLE RESPONSES BY SOURCE OF HE/iDINGS^ 

Complete Headings 
Direct 



OK 


P/S 


AB 


OTHER 

FORM 


INV 


TOTAL 


Card 


30 


4 


• • 


2 


5 


41 


Question 5 


80 


3 


3 


1 6 


• • 


93 


SH Question 


41 


4 


• • 


1 1 


2 


49 


Total 


151 


11 


3 


2 9 


7 


183 








Cross 


i-reference 








OK 


P/S 


AB 


OTHER 

FORM 


INV 


TOTAL 


Card 


21 


1 


•' • 


2 


1 


25 


Question 5 


40 


5 


• • 


5 


1 


51 


SH Question 


54 


• • 


• • 


3 


1 


58 


Total 


115 


6 


• • 


10 


3 


134 




Partial Headings (NS and WS 


combined) 










Direct 








MOK 


MP/S 


MAB 


MSP M FORM 


TOTAL 


Card 


14 


2 


• • 


• • 


4 


20 


Question 5 


29 


4 


1 


1 


6 


41 


SH Question 


36 


14 


• • 


• • 


0 


40 


Total 


79 


10 


1 


1 


10 


101 








Cross-reference 








MOK 


MP/S 


MAB 


MSP M FORM 


TOTAL 


Card 


7 


• • 


• • 


• • 


1 


8 


Question 5 


18 


• • 


1 


• • 


4 


23 


SH Question 


37 


1 


• • 


• • 


4 


42 


Total 


62 


1 


1 


• • 


9 


73 


^For explanation 


of 


sources 


, see 


p. 1. 







AN ANALYSIS OF NON-EXACT TITLE DATA 
William A. Hinkley 



Library users, wishing to retrieve a specific work known by 
them to exist, can enter the card catalog as it now exists with author, 
title, or subject information. The subject heading is an addition made 
by the subject cataloger to a given book, but the author-title citation 
is an integral part of the book and essentially unique to it. It can 
be remembered through rote memory without any intervening concept - 
formation process. If the user must rely on his memory of previous 
exposure to the desired book, it is not unlikely that he has faulty and 
imprecise information about the details of the author-title citation. 

In the experimental situation described above, experimentees 
were asked to recall what they could about the books they had seen at 
an earlier time. Very few experimentees (20.9 per cent) offered any 
author information, but when they did, they were usually correct. Most 
experimentees {82.7 per cent) offered title responses, but only a small 
percentage of them (22,7) were word perfect. Since only a few experi- 
mentees offered inexact author information, there is little potential 
for improvement of retrieval by author. However, a substantial per- 
centage of experimentees thought they remembered and consequently 
offered non-exact title responses. This situation suggests that a close 
analysis of non-exact title responses be carried out to determine how 
many of them are partially correct and potentially useful. 

There are several ways in which this non-exact information might 
be utilized. It will be important to keep these methods in mind as the 
data are analyzed. A concordance entry to the individual words in titles 
could be provided by a permuted title list or by added entries in the 
existing card catalog for individual words in titles. A title catalog 
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stored in machine readable form could be searched with the specification 
that certain remembered words co-occur in the retrieved title. 

The following analysis will concern itself with what is remem- 
bered correctly in non-exact responses. The utility of this remembered 
information is influenced by several other important but separate con- 
siderations. For example, there might be too many entries at one 
alphabetical word position in a permuted title list or card catalog 
with added title word entries. Further, the retrieval value of extra 
title word entries might be canceled out by the bulk addition to the size 
of a card catalog or permuted list. Although these questions should be 
considered when assessing the potential utility of non-exact responses 
to the correct title, they are separate from an .evaluation of book title 
memorability, and are not dealt with in this paper. 

Non-exact title responses which were experimentally derived can 
be used as simulated queries to a real card catalog, or to any improved 
retrieval system using concordance or coordinate access. To evaluate 
the usefulness of these non-exact queries, it will be first necessary 
to determine how many exact titles are retrievable from a current card 
catalog, when it is entered with this non-exact information. Two assump- 
tions must be made. First, it must be assumed that users can recognize 
the correct title once their non-exact query has allowed them to narrow 
the search to a few titles. There is evidence to suggest that this is 
reasonable. Seventy-seven per cent of the experimentees in the book- 
memory experiment were able to recognize from long author-title citation 
lists the citations for the books that they had seen before. 

Secondly, the arrangement of the card catalog must be considered 
in order to determine the minimum information that is required to locate 
items in it successfully. There is no easily specified minimum informa- 
l;j[on requirement for access to the alphabetical file of the card catalog. 
This minimum is a function of file size, number of entries under one 
word, and the number of cards a user will want to search. Consequently, 
it is necessary to make the following general assumption. 

If the first two words of a non-exact query correctly match the 
'first two words in the title sought, then it is assumed here that the 
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query contains enough information for retrieval* Queries meeting this 
condition will have, by definition, a high retrieval value. Retrieval 
can probably also take place if the first word of the query matches 
exactly the first word of the desired title, but in this case the task 
is more difficult. There will be more entries to go through and more 
confusion will result from the fact that titles are not in a separate 
file but interspersed with author and subject entries. Queries which 
ifieet the first word match condition will be said to have low retrieval 
value. 

The group of queries with low retrieval value is also defined 
to contain some even less exact queries. If the first five letters of 
the first word of the query match the first five letters in the first 
word of the title, the query was counted as having a low retrieval value. 
The library user would have to search through an even larger file than 
in the above cases but the recognition assumption would probably assure 
retrieval. The sixteen queries that had five-letter low retrieval 
value differed in their first words from the first words in the exact 
title by only slight variations in grammatical endings. On only one 
occasion did the first query word differ significantly from the correct 
one. The experimentee offered "autocontrol" in response to the correct 
word "autoconditioning." Thus mistakes in the first word, if they are 
made, are probably either complete word changes or very nearly corrett. 

Table 1 shows the distribution of retrievable responses broken 
down by the number of words in the exact tit.:.e* First, the table shows 
certain facts about the sample. There is a ]>reponderance of short titles, 
five words or less. The sample was made up of monographs in psychology 
which traditionally have shorter titles than journal articles or reports. 
There are a fair number of titles in each "number of words" category. 

The modal title length was three words with titles of one, two, four, 
and five words each occurring about one-half as frequently. There are 
approximately three responses for each title in the sample. 

It should be noted from Table 1 that there are 446 responses, 
which figure does not include seventy-eight occurrences of the null 
response, that is an empty response, to the question, "What is the title 
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of the book?'* Because of the format of the questionnaire, some experi- 

mentees gave two responses to the title if they thought their first 

guess was incorrect. Each of these responses was treated as a separate 

response. It was assumed that each non-exact response was a separate 

problem for the interrogated file, whether or not it came from one person. 

There are approximately eighty double responses in the 446 responses 
1 

tallied. 

TABLE 1 



DISTRIBUTION OF CONVENTIONALLY RETRIEVABLE RESPONSES BY 
NUMBER OF WORDS IN THE CORRECT TITLE 





a 

Number of Words in Title 




Retrieval 
Utility of 


1 


2 


3 


4 


5 


6 


7 


8 


Total 


Responses 


(15) 


(21) 


(40) 


(25) 


(20) 


(4) 


(3) 


(6) 




Exact title match 


20 


24 


25 


19 


3 


1 


-■ 


- 


92 


High retrievability 
(first two words) 






6 


12 


3 


4 


- 


3 


28 


Low retrievability 
(first word) 


10 


7 


8 


5 


9 


2 


- 


1 


42 


No retrieval 
value 


20 


45 


87 


64 


39 


2 


4 


23 


284 


Total 


50 


76 


126 


100 


54 


9 


4 


27 


446 



^Numbers in parenthesis indicate the number of titleit in each 
word -length category in the sample. 



Table 1 shows the number of responses that wer <2 word perfect in 
each "number of words" class. Responses to titles with only one word were 



^Data was gathered for this analysis before final verification ^ 

of the data base. Consequently this analysis differ ::rom the final 
tallies reported above in one or two cases. 
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word perfect in 40 per cent of cases (20/50); 33 per cent of responses 
to titles with two words were word perfect; 20 per cent of responses to 
titles with three and four words were word perfect. A much smaller 
percentage of the responses to titles with five or more words are exact 
but the numbers in the sample are small. As might be expected the chance 
for error gee' up sharply as the number of words in the title increases 
and the added complexity taxes memory more. 

Tsble 1 also shows the seventy non-exact responses having high 
or low retrieval value. Thus 19.5 per cent (70/354) of all non-exact 
title responses are retrievable from the conventional card catalog. 

When this figure is added to the ninety-two correct responses we have 162 
responses that are retrievable. This figure is 36 per cent of the total 
number of exact and non-exact responses in the experimental data. It is 
31 per cent of all responses including null responses. Thus, approxi- 
mately one-third of the time, the response to the question, "What is the 
title of the book you saw?" is retrievable from a conventional catalog. 

The responses with high and low retrieval value have several 
characteristics worth noting. Of the twenty-eight responses with high 
yalue, eighteen match the first two words of the correct title, the 
minimum number for the "high" designation. The remaining ten responses 
match the first three words, and none match the first four or more words. 
The forty-two responses with low retrieval value include sixteen responses 
that were not exact matches in their first words, but were at least five 
letter matches. Six of these five-letter matches were in response to 
one word titles. Two of these responses were exact matches to three 
word titles except for a variant in the last syllable of the first word. 

Up to this point the central concern has been retrieval from a 
conventional card catalog using non-exact title information. In order 
to evaluate the worth of non-exact title information when used with a 
permuted title list or computer coordinate search of the complete title 
text of a book catalog, it will be necessary to change some of the above 
assumptions. The first assumption, that users can recognize a title they 
had seen before from a short list of titles, will hold. The second assump- 
tion must be changed, however. Retrieval can take place with these new 
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file arrangements and access methods if the user remembers correctly 
any words in the title j the requirement that he remember the first word 
of the title, which can be called a positional requirement, no longer 
holds. Specific assumptions about how much of a non-exact response must 
be correct for retrieval are beyond the scope of this paper and will 
not be made. Instead the following analysis will demonstrate a number 
of observations about the qualities and quantities of memory in non- 
exact responses. 

First it will be necessary to show how many words in each non- 
exact response exactly match words occurring in the correct title.- 
Table 2 divides the 446 exact and non-exact responses according to 
(l) the number of words in each response that exactly match the title 
words, and (2) the number of words in the correct title. The distribu- 
tion of the seventy responses retrievable from a conventional card catalog 
is also included. The notation gives the number of non-exact responses 
retrievable followed by an "L" or "H" to indicate high or low retrieval 
value. 

TABLE 2 

DISTRIBUTION OF RESPONSES BY THE NUMBER OF WORDS IN EACH 
RESPONSE THAT EXACTLY MATCH TITLE WORDS AND BY NUMBER OF 

WORDS IN THE CORRECT TITLE 

[Distribution of high (H) and low (L) valued responses] 



No. of Words in 
Response Exactly 
Matching Title Words 


Number of Words in the Title 


1 


2 


3 


4 


5 


6 


7 


8 


0 


21 


23 


45 


18 


14 


1 




5 


1 


29 


26 


31 


34 


12 




2 


9 




(lOL) 


(7L) 


(4L) 


(5L) 


(2L) 






(IL) 


2 




27 


19 


16 


13 




1 


2 




' 




(3H) 


(3H) 


(2L) 














(2L) 












3 






31 


12 


4 


3 


1 


5 








(3H) 


(9H) 


(IH) 


(IH) 












(2L) 




(2L) 


(IL) 






4 








20 


7 


3 




2 












(IH) 


(3H) 




(IH) 












(2L) 








5 










4 


1 




3 












(IH) 


(IL) 




(IH) 












(IL) 








6 












1 




1 


















(IH) 



[ *- 
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Table 3 demonstrates the change in matching performance when 
matches on the words "of," and "and," and "the" (when not the first word 
in the title) are subtracted from Table 2. It is obvious that these 
words are of little retrieval value in concordance-type access. They 
are very frequent in title text. These easy and useless matches seldom 
occur in one and two-word titles; syntax prevents it. There are a large 
number of these matches in the three-word titles .because of the fre- 
quency of the title constructions, (noun) and (noun); (noun) of (noun). 
Therefore the thirty-one one-word matches reported in Table 2 under 
the three-word title column is reduced in Table 3 by thirteen of -and -the 
matches. The nineteen two-word matches are reduced by fourteen of-and-the 
matches, although the of-and-the might be of some retrieval use once the 
main word had been discovered. 



TABLE 3 

DISTRIBUTION OF RESPONSES BY NUMBER OF INTERNAL 
MATCHES MINUS OF-AND-THE MATCHES 



No. of Words (not 
of-and-the) Exactly 
Matching Title Words 




Number 


of Words in Title 




Total 


1 


2 


3 


4 


5 


6 


7 


8 


0 


21 


23 


58 


19 


15 


1 




9 


146 


1 


29 


26 


32 


44 


15 




2 


6 


154 

• 


2 




27 


8 


12 


13 


2 


2 


3 


67 


3 






28^ 


6 


6 


4 


0 


5 


49 


4 








19^ 


2 


1 




3 


25- 


5 










3^ 


0 




1 


4 


6 


1 


1 








1^ 






1 



^atch on of-and-the not subtracted if response was word 

perfect. 



It is immediately apparent from Table 3 that a large number of 
titles contain zero or one matches. There are 146 zero matches including 
nineteen one-word of-and-the matches. There 154 responses with one-word 



matches including the two-word matches with one of -and -the and the twenty 
correct responses to One-word titles. There are 146 responses with at least 
two good matches, of which seventy-two are correct responses. Therefore, 
of the 354 non-exact responses, 41 per cent (146) contain zero exact 
and good matches, and 59 per cent (208) contain one or more. Twenty per 
cent (74) of all non-exact responses contain at least two good word 
matches. However, forty-one of these responses are retrievable because 
they contain positional information, i.e., stating correctly the first 
word or first two words of the title. 



Table 4 divides the experimentally derived responses into cate- 
gories of their usefulness when applied to several retrieval situations. 
The percentage breakdown of non-exact responses is given at the right. 
Twenty per cent of the responses are retrievable from an existing card 
catalog under the assumed retrieval specifications outlined above. 

Forty per cent contain no internal matches, i.e., no response words 
matching words in the correct title. Forty per cent contain one or 
more internal matches, but only one-fourth (33/138) of these contain 
more than one internal match. This fact somewhat invalidates arguments 
for a computer search of a title list with a retrieval specification 
that two words from the response co-occur in the retrieved title. Some 
of the responses in the retrievable category (II) contain two or more 
internal matches but there would be little point in searching^ for them 



TABLE 4 



DISTRIBUTION OF NON-EXACT RESPONSES 

Number Percentage 



I. Retrievable from card catalog 

First two words correct 
First word correct 
First five letters correct 

II. Not retrievable from card catalog 

a 

One -word match 

Two or more word match 

No match 



28 




7.8 




26 




7.4 




16 




4.4 






70 




19.6 


105 




29.6 




33 




9.4 




146 




41.4 






284 ^ 




80.4 




354 




100.0% 



^Does not include a match on of-and-the. 
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by computer when they are already retrievable with a conventional card 
catalog. 

Constructing retrieval specifications for a coordinate search 
„ould be further complicated by the fact that it would be difficult to 
identify which words in the response were correct match words. Table 
demonstrates the number of responses containing superfluous words. Each 
boxhead indicates the number of responses in each match category; the 
column below it the number of responses containing more words than the 
correct title has. In the zero match category, the numbers in the row 
indicate the number of responses with two or more words. The 127 zero 
match cases despite complete failure at matching were able to elicit 
ninety-eight responses with at least two words. The shorter titles 
seem to elicit more responses with superfluous words than the longer 
titles. However, overall, there are relatively few responses containing 

more words than their correct titles. 



TABLE 5 

distribution of responses containing at least 
Im mm word than in correct t™ 




two or more words. 



1 

TITLE, SUBTITLE AND TABLE 03? CONTENTS AS SOURCES OF INDEX WORDS 

Neil A. Radford 

Generally speaking, author, title and subject are the three 
possible approaches to using the contents of a library catalog, and it 
has always been assumed that the catalog user will be able to specify 
at least one of these. Every reference librarian has encountered the 
problem of the reader who has used a particular non-fiction book in 
the library before and now wishes to see it again, but has completely 
forgotten its author and title. In such a case the librarian usually 
assists the reader in a subject-oriented search at the catalog or in 
an appropriate bibliography, and in all but the smallest libraries, a 
considerable amount of time is spent plodding through the various subject 
entries, hoping to see a likely -looking author or title. 

With present library cataloging practice, a title approach is 
possible only if the user has accurate knowledge of the whole title or 
at least the first part of it, and a subject approach is usually con- 
sidered to be potentially more fruitful when the title cannot be remem- 
bered because the reader at least knows the subject matter of the book 
in a general way, and can often define it more specifically under ques- 
tioning from the librarian. What seems seldom to be realized is that 
because the majority of non-fiction books have titles which are closely 
tied to their subject matter, it is possible that, in being questioned 
about the book*s subject, the forgetful library user might be able to 
recall enough significant title words (without realizing it) to permit 
a faster and more. fruitful catalog search by title. 



^The experiment reported here was originally carried out as an 
exercise in computer programming, rather than an 

ing previously seen books by means of index words taken from thexr titles, 
subtitles and tables of contents. 
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Several points arise here. To what extent is such questioning 
likely to bring forth significaiht words from the title? Further, to 
what extent is it likely to bring forth words from other parts of the 
book which, although not presently indexed in the library's catalog, 
could conceivably be handled in a mechanical system to improve the chances 
of retrieval by a subject approach? 

The Graduate Library School of the University of Chicago has 
recently completed an experiment designed to discover what additional 
characteristics of books, besides those presently included in library 
catalogs, could be added to the catalog entry to facilitate retrieval 
of wanted materials which have been previously consulted, and the data 
collected allow some conclusions to be made respecting the specific points 
raised in the preceding paragraph. The details of the experiment are de-^^ 
scribed in "Memorability of Book Characteristics: An Experimental Study," 

but are presented here briefly. A number of participants were asked to 
select several books which interested them from a controlled collection 
drawn from the University Library's psychology section, and to examine 
these books and write comments about them. Without being told the reason 
for the experiment, the participants were asked to return two weeks later, 
at which time they were requested to write down everything they could 
remember about the books they had previously consulted. Thus their 
recollections are probably quite close to those of a library patron who 
reads a book which interests him, without expecting to need it again, 
and later comes back seeking the same item. Among other things, partic- 
ipants in the second, or "recall,” stage of the experiment were asked to 
write down the author and title of each of the books they had previously 
examined, or, if the exact author and title could not be remembered, then 
as much of them as they could manage. They were also asked to write ^ 
down the subject headings under which they would look in the Library's 
catalog if they were trying to find the book concerned, and it was this 
information which made possible the study reported here. 

The records of responses were examined and those instances in 
which the participants were completely unable to remember anything about 
the books’ titles but had given one or more subject headings, were selected 
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All responses pertaining to books with the word "psychology" in their 
titles, or responses including the word "psychology" as a subject heading 
were excluded from this selection on the grounds that, as the experi- 
mental collection was drawn from the Library's psychology collection, 
all the books in it came under the very broad heading of "psychology," 
and this general term would not be a. useful means of identifying or 
retrieving a specific book. An additional few responses were also 
excluded from the analysis because the books to which they pertained 
were not immediately available for inspection. 

The twenty-four responses finally selected covered, in all, 
twenty different books. These twenty books were then examined, and what 
might best be called "key words" were noted down from their titles, sub- 
titles and contents pages. Generally speaking, the words recorded for 
each book comprised all nouns from the title and subtitle (if any) and 
all important nouns or ideas from the list of contents. Some discretion 
had to be used in selecting words from the contents lists of a few books 
which defined in great detail what each chapter and each section of each 
chapter was concerned with, and in a few cases where there were literally 
dozens of possible words, none of which corresponded with the subject 
words given by the respondent, all were ignored for the sake of simplic- 
ity. This may have reduced the number of incorrect matches in the course 
of the experiment, but the number of correct matches was not affected 
because the words omitted did not correspond with the words specified 
by those respondents who were attempting to retrieve the books in question, 
so no match would have been made anyway. 

One other comment on the nature of the index words needs to be 
made, namely that, for the sake of simplicity again, variations in the 
form of a word were grouped with the word itself. For example, the words 
"child," "children," and "childhood" were all converted to the form 
"child," both in the list of words obtained from the books and in the 
subject words given by the respondents. This type of "standardization" 
parallels the real library situation where, to ensure uniformity, a 
thesaurus such as a subject-headings list determines which words and 
which forms of words are "permissible" and words from books and words 
from library patrons have to conform to its rules. 
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In twenty-one of the twenty-four searches, the matching of index 
words with response words resulted in retrieval of the author and title 
of the book which the experiment participant had been trying to recall. 

In other words, in 88 per cent of the searches, matching the subject 
words remembered by the participants in the experiment with actual words 
taken from the titles, subtitles and/or lists of contents, resulted in 
the correct books being found, even though the participants claimed ho 
be quite unable to remember anything at all concerning these features 
of the books, and had given these words in response to a request for 
subject headings. Although the sample size was small, a 95 per cent 
confidence interval constructed around the results gives a range of be- 
tween 6i per cent and 97 per cent, which means, roughly, that the odds 
are twenty to one against the possibility that the true percentage, which 
would have been obtained by increasing the sample size indefinitely, lies 
outside this range. Therefore, although this confidence interval may 
seem a large one. it at least shows that the sample is not so small as 
to be useless, since it is reasonably assured that in dealing with a 
vastly larger collection, the percentage of successful searches would 

be no less than 67 per cent. 

In all, the twenty-four searches resulted in a total of sixty- 
nine books being retrieved, forty of which were "false drops" and twenty- 
nine correct matches. This last figure is larger than the number of 
successful searches because of the fact that many of the experiment 
participants had specified more than one "subject word." which sometimes 
resulted in the correct book for a particular person being retrieved 
more than once, as his words were processed in turn. An average of 
two and one-half different titles were retrieved per search, and if we 
define the "retrieval rate" as being the percentage of the total collec- 
tion which was retrieved, we find that the retrieval rate for the experi- 
ment averaged 12 1/2 per cent. We may therefore assume that a similar 
proportion of a larger collection would be retrieved in a process of this 

nature. 

The sources of the index words which led to the twenty-nine 
retrievals are interesting. Ten, or 34 per cent, were found through 
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index words taken from the titles of the books, six, or 21 per cent, 
through index words taken from the subtitles (making 55 per cent for 
title and subtitle together), and thirteen or 45 per cent, through 
index words taken from the list of contents. 

In an actual library situation using a computer for searching 
and retrieval, the program could be designed so as to minimize the in- 
convenience caused by "false drops," by ranking the retrieved citations 
in order of their likelihood of being the wanted item, based on the 
number of times they had been found. For example, if four subject words 
were given by a library user, those citations which matched with all 
four words would be ranked "most likely," those with three matches would 
be next in order, and so on. In this way the task of searching a long 
list of titles with many false drops could be expected to be made easier, 
since the user could examine the most promising items first. 

Two tentative conclusions can be drawn from these results. First, 
because requests for subject headings will frequently produce words from 
any part of the title, a permuted title index is likely to be consider- 
ably more useful for finding previously seen books than a "traditional" 
library catalog with title entries, because it permits the user to 
search on any "key word" in the title. Present library catalogs permit 
a fruitful title search only if the first important title word is known, 
and in only two of the twenty- four instances in the experiment were 
respondents able to give the first important title word, although eight 
were able to give other important words from the title. 

Second, and probably more important, is the conclusion that 
taking index words only from the title of the book is significantly less 
successful in retrieving books previously seen than is an index, of 
whatever kind, which is comprised of "key words" taken from the title, 
subtitle (if any) and the list of contents. Two-thirds of the success- 
ful searches in the experiment for previously seen books owed their 
success entirely to the inclusion in the corpus of index words, of words 
taken from the subtitle and/or list of contents of the books concerned. 
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The added indexing effort is likely to be substantial if these extra 
sources of "key words" are to be tapped, but on the basis of the results 
reported here, the increased number of successful searches io so sub- 
stantial as to warrant serious consideration being given to their 
inc lusion. 
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APPENDIX 

THE *'1NDEX WORDS'* AND "REQUEST WORDS*' USE D IN THE EXPERIMENT 



TITLE AND SUBTITLE 
Guidance and counseling in groups 



Rowan tree and red thread : a 
Scottish witchcraft miscellaney of 
tales, legends and ballads to- 
gether with a description of the 
witches* rites and ceremonies 



A parenthesis in eternity 



An application of psychoanalysis 
to education 

The ape and the child: a study of 
enrivonmental influence upon 
early behavior 



Stolen fire: a study of genius 

Toward understanding human 
personality 



Stress situations 



The growth of reason: a study of 
the role of verbal activity in the 
growth of the structure of the 
human mind 



WORDS FROM 
CONTENT S LIST 
Living 
Learning 
Vocation 
Education 
Personalia 



Mysticism 

God 

Reality 

Illusion 

Consciousness 

Mind 

Spirit 

Universe 



Prejudice 
Perspective 
Ego 



RESPONSE WORDS 

Counseling* 
Psychotherapy 
Psychoanalysis 
Illness 

Scotland* 
Witchcraft* 



Mysticism'^ 

Christianity 



Pre judice’f 
Emotion 

Monkey 
Child* 



Anthropology 
Child* 



Eros 

Prometheus 
Psychotherapy 
Life 

Motivation 
Child 
Habits 
Frustration 
Failure 
Illness 
Catastrophe 
Marriage 
Fertility 
Sterility 
Divorce 
Death 
Suicide 
Intelligence 
Child 
Symbolism 
Meaning 
Syntax 
Abstraction 
Logic 



Persona lity- 



Frus tration’f 
Failure* 
Fear 
Phobia 



Logic* 

Behavior 



* 



^Responsp. words resulting in a successful match with words from 
title, subtitle or list of contents. 
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APPENDIX (Continued) 



TITLE AND SUBTITLE 



WORDS FROM 



RESPONSE WORDS 



Existence: a new dimension in 
psychiatry and psychology 


Exis tentialism 

Psychotherapy 

Phenomenology 

Schizophrenia 

Depression 

Aesthesiology 

Hallucinations 

Insanitv 


Ontology 

Metaphysics 


The rape of the mind : the psychology 
of thought control, menticide and 
brainwashing 


Confession 

Fear 

Torture 


Brainwashing* 


Love against hate 


Frustration 

Child 

Woman 

Work 

Play 

Faith 

Hope 


Faith* 

Hope* 

Love* 


Behavior 

Adiustment 


Behavior 

Play* 

Recreation 


Children who cannot read: the analysis 
of reading disabilities and the use of 
diagnostic tests in the instruction of 
retarded readers 




Reading* 

Education 

Literacy 


Ifyth and guilt: the crime and 
punishment of mankind 


Culture 

Conscience 

Memory 

Science 

Religion 


Guilt* 

Psychoanalysis 
Primitive man 


Guilt* 

Religion* 

Superego 


Extra- sensory perception 


Parapsychology 


Parapsychology* 

ESP* 


Science and human behavior 


Reflexes 

Environment 

Groups 


Behavior* 
Political scien 


The origin of consciousness: an 
attempt to conceive the mind as a 
oroduct of evolution ^ ^ 


Body 

Psyche 


Mind* 

Evolution* 


Order of birth, parent-age and 
intelligence 


IQ 

Child 




IQ* 

Child* 
Mental age 


Inside the black room 


Sensory 

deprivation 

Abilities 

Learning 


Sensory 

deprivation* 


Minor mental maladjustments in 
normal people: a casebook for the 
use of students of mental hygiene, 
psychology, education, child 
development, sociology, and the 
formation of personality traits 




Neurosis 

Trauma 



* 



'^Response words resulting in a successful match with words from 
title, subtitle or list of contents. 



THE POTENTIAL USEFULNESS OF CATALOG ACCESS POINTS 
OTHER THAN AUTHOR, TITLE AND SUBJECT 

William S. Cooper 

Most of us have had, at some time or other, the frustrating 
experience of being able to remember quite vividly the general physical 
appearance of some book we have read, but of being unable to recall . 
either its author it its exact title. Professors complain that what 
their students remember best about a textbook is the color of its cover. 
A book owner is often able to remember what condition one of his books 
is in, even though its author and title may have for the moment escaped 
his memory. These commonplace observations suggest that there is often 
a considerable amount of miscellaneous ’’nonstandard" information con- 
cerning a document which is more memorable to one who has had contact 
with the document than the "standard" author-title-subject information 
is. This paper is addressed to the question of whether it might be 
worthwhile to attempt to make use of such nonstandard information in 
the design of future library catalogs and retrieval systems. 

1. Nonstandard Information 

Let us agree to use the phrase "nonstandard information" as a 
cover term for all those types of information about documents which 
could conceivably be helpful in specifying a wanted document, but which 
cannot be used to look up a document in the catalogs found in present- 
day libraries. Since present-day libraries normally provide nothing 
but author, title, and subject catalogs, this includes virtually all 
types of data about documents except for author- title-subject informa- 
tion. Data such as the date of publication of a document, its length 
(number of pages), and an indication of the presence or absence of 
illustrations in it are usually to be found in present-day catalogs; 
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but under our definition these must be regarded as nonstandard types 
of information, for such data cannot be used for lookup purposes 
as long as catalogs are alphabetized only according to author, title, 
or subject. Of course, such physical features of a document as its 
cover color and design are definitely "nonstandard," since they do not 
appear in today’s catalogs in any form. 

All nonstandard information represents an as yet untapped re- 
source for aiding the retrieval process. To illustrate the possibili- 
ties, suppose that a patron of a library or infofmation center desires 
a particular document with which he has had previous contact. He may 
be unable to remember the author or title of his document with the 
accuracy and completeness needed to look it up in a conventional author- 
title catalog. He may also be unable to specify the subject area in 
terms of a heading which would permit a quick lookup in the subject 
catalog. But it is quite possible that he can nevertheless state with 
assurance that the document in question is, say, large, green, m good 
condition, recently published, a translation from the German, and well 
illustrated. Here is a case where the remembered nonstandard information 
has the potential for being extremely useful, for surely there could not 
be very many documents in the collection answering to this description. 

If these few documents (possibly just one) could be retrieved and 
assembled for the requester’s inspection, he would presumably have no 
trouble in picking out the needed one from among them. Examples could 
also be given in which the nonstandard information might not be suffi- 
cient in and of itself, but would nevertheless become sufficient if it 
could be combined with whatever shreds of standard information are 

remembered. 

The possibilities inherent in nonstandard information have never 
been seriously exploited in conventional libraries. There are good 
reasons for this. It seems to be the nature of nonstandard information 
to consist of a variety of "weak clues" rather than just one "strong 
clue." For example, the information that a book has a green cover is 
a "weak clue" in the sense that there may be thousands of books in the 
collection with green covers. It follows that the clue "green-covered" 
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would be by itself quite inadequate for retrieval, and it is only by 
combining a number of such weak clues that one might hope to narrow 
down the set of candidate documents to a helpful extent. Now, to 
construct and maintain a conventional file such as a card catalog for 
looking up combinations of weak clues would be a difficult task indeed. 
The file would have to be alphabetized for lookup first on one type of 
information, say color, and then within color on, say, size, within size 
on on something else, and so forth through all the various nonstandard 
clue types. But one file would not be enough; the requester who has 
forgotten the color of his book but remembers other nonstandard informa- 
tion would need a different file alphabetized first on some other clue 
type which he is able to remember. Continuing this line of thinking, 
it becomes apparent that one would need almost as many different files 
as there are permutations of clue types --a number large enough to raise 
doubts as to whether there would be any room left for books in the 
library after all the catalogs were installed. And as if this problem 
were not serious enough, we shall see later that any combination of 
several weak nonstandard clues reported from memory is likely to contain 
at least one clue that is in error, a circumstance which complicates 
the problem of catalog design immeasurably. 

For these reasons it seems impractical to try to exploit non- 
standard information by conventional means. However, it does not 
necessarily follow that it could not be exploited with the help of 
computers. It is unthinkable that a library user should have to scan 
all or even a substantial fraction of a complete holdings file to track 
down his document, but it is not unthinkable that a computer might do 
so. With an automated catalog or retrieval system, it may well be 
within the realm of practicality to perform searches for any arbitrary 
combination of remembered nonstandard clues. Moreover, the problem of 
partially mistaken information is not necessarily insurmountable if the 
use of computers is contemplated. Should the wanted document turn out 
not to be among the retrieved documents fulfilling his description, it 
must then be among some set of documents which almos_t matches his de- 
scription, assuming that most of the data in his description is correct. 
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The nearly matching documents which are most likely to include the wanted 
document could be retrieved for the requester’s inspection, then the 
next most likely, and so on until either the requester finds what he 
was looking for, or else the probability of doing so in a reasonable 
amount of time drops too low to justify continued searching. This plan 
would require the computer to make some sophisticated probability com- 
putations, it is true, but the retrieval process would not necessarily 
be significantly slowed down because of this. Numeric computation is 
just what computers are best at. 



It is not our purpose here to describe exactly how a computer 
could be prograiraned to perform searches on the basis of possibly mis- 
taken clues, except insofar as we shall have occasion in a later section 
to set forth for a special case the probability calculations which the 
computer would be required to make. The questions which should claim 
prior attention to all programming considerations are the following: 
Assuming that appropriate retrieval mechanisms were somehow made avail- 
able for exploiting nonstandard clues to the maximum possible extent, 
how great an improvement in retrieval performance could one expect? 
Specifically, are there a significant number of potential library patrons 
who possess nonstandard information about the documents they need, and 
who would make use of whatever future facilities might be provided to 
exploit this information? And even more important, would these people 
tend to remember enough nonstandard information, and remember it 
correctly enough, to make the potential improvement in retrieval results 
worth pursuing. These questions will be the focus of attention in what 

follows. 




How great is the need for a means of allowing library patrons 
to use nonstandard information in their search for documents they want? 
Unfortunately it is impossible to gauge this need by questioning users 
in present-day libraries. The reason for this is that present-day 
libraries can do little or nothing to help a potential patron whose 
only information is of the nonstandard variety, and so such patrons, 
knowing this, stay at home. The best that can be done with current 
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user surveys is to make some rather oblique inferences about the probable 
number of potential library users who find themselves in the position 
of having nonstandard information but no way to exploit it. 

In discussing questions of document retrieval it is helpful to 
distinguish between what is usually called subject retrieval and what 
we shall here call specific work retrieval . In the subject retrieval 
situation, the library user wishes to locate works on a topic or sub- 
ject area of interest to him, but he does not necessarily have any prior 
familiarity with the works, nor for that matter even any advance assur- 
ance of their existence. The specific work retrieval situation is 
quite different in that the requester already knows of the existence of 
some particular document which he wants. Clearly, nonstandard informa- 
tion about the wanted document will enter the picture chiefly in connection 
with specific work retrieval, for the user will normally be able to 
specify detailed physical details only in connection with particular 
works with which he has had previous contact. 

Since nonstandard information has potential value mainly for 
specific work retrieval, it is pertinent to ask whether specific work 
retrieval s,earches form a substantial part of all catalog searches made 
in present-day libraries. A number of catalog use studies provide 
evidence that they do. In E. Montague's recent survey of card catalog 
use studies [1]? a summary is presented which includes fifteen studies 
that give clear statements of the proportion of catalog searches ob- 
served which were of the specific work type. The fifteen percentages 
for the proportion of specific work searches ranged from 41.8 per cent 
(for a public library system) up to 90 per cent (for a specialized 
library), the median figure being 60 per cent. Two investigators com- 
mented on a tendency for the proportion of specific work searches to 
be higher in libraries where the educational level of the population 
was high. The picture which emerges is that the preponderance of catalog 
usage today has as its purpose the locating of specific known works, and 
that this is especially true in libraries which serve a highly educated 
or highly specialized community of users. Of course, proposed improve- 
ments in specific work retrieval facilities such as thdse considered 
in this paper would well increase even further the proportion of specific 
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work searches made in future libraries, for the proportion of specific 
work searches versus subject searches reflects not only user needs but 
also the adequacy of the facilities made available for specific work 
retrieval and subject retrieval. 

Another issue of interest is the number of potential searchers 
for a specific document who have actually had contact with the document 
itself. Those who have not are poor prospects as users of future 
facilities to exploit nonstandard information for searches based ex- 
clusively on a citation or bibliography are likely to be completed on 
the basis of author-title information alone, assuming they can be success- 
fully completed it at all. A recent survey made by M. Blackburn [2] 
provides some idea of the proportion of specific work searches whose 
purpose was to locate documents actually read or at least seen by 
searchers. In Blackburn's survey. 126 patrons of the University of 
Chicago library were stopped and questioned as they reached for a drawer 
of the main catalog. One hundred of these 126 turned out to be attempting 
to locate a specific work, confirming the previously mentioned findings 
to the effect that a majority of present-day catalog users seek known 
items. Of the 100 seeking a specific work, twenty-nine stated that 
they had previously seen or used the work they were looking for, and 
an additional thirteen who had never actually seen their document had 
nevertheless been given a verbal recommendation of it by someone who had. 
This makes a proportion of at least 29 per cent and possibly somewhat more 
of specific work searchers who would have been in a position to supply 
nonstandard information from their own personal memory or that of their 
colleagues. This finding is admittedly based on a small sample, and is 
not necessarily generalizable to other libraries with different user 
populations, but it serves to suggest at least that the number of searchers 

who possess nonstandard information is not insignificant. Moreover, the 

„nH<a■rs^ates to some unknown extent the 
29 per cent figure presumably understates 

. t-vr\aT on to suddIv nonstandard informa 
number of potential patrons in a position to supply n 

■ tion, for only those who could supply a sufficiency of standard informa- 
tion’ in addition to the nonstandard ever reached the library catalog to 

be interviewed. 
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The reader is referred to [2] for further details concerning the 
amount and types of nonstandard information which users of the University 
of Chicago library catalog tend to remember, as well as the accuracy of 
their recollections. It appears that a considerable amount and variety 
of nonstandard information, some of it accurate and some inaccurate, is 
in the users' memories. However, the ultimate significance of findings 
from user surveys for the design of future catalogs is unclear for the 
reason already mentioned, which is that what library patrons can remember 
at today's catalogs is not necessarily indicative of what the potential 
users of tomorrow's improved catalogs might be able to remember if 
future catalog facilities were designed to allow use to be made of non- 
standard information. 

3. The University of Chicago Memory Experiment 

In view of the limitations of the user survey method, it was 
decided to conduct a special experiment to gain insight into what types 
of information tend to be remembered most often and most accurately about 
books commonly found on library shelves. In this "memory experiment, 
as we shall call it, human subjects were to be exposed to books and 
after a suitable lapse of time were to be questioned to determine what 
they could remember about the books. A memory experiment can, if properly 
designed, control to a large extent such problems as variability in the 
duration of the subjects' exposure to the books and also variability in 
their browsing or reading environment at the time of exposure. The 
experimental approach has its drawbacks as well, for one can never be 
certain that the length of exposure, experimental environment, motiva- 
tion, time lapse between exposure and questioning, and other conditions 
of the experiment are truly representative of real-life conditions. 
Nevertheless the advantages of the experimental approach were felt to 
more than justify it as a complement to user survey techniques, and so 
the experiment was undertaken. 

t 

The experimental design is fully described elsewhere [3, 4]. 
However, the features of the experiment which are essential for present 
purposes are easily sketched. Briefly, a large number of subjects were 
allowed to select and read from books which interested them, but with 
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which they had had no prior acquaintance. Two weeks later, each subject 
was asked to fill out, from memory a mimeographed questionnaire about 
each book from which he had read. The questionnaire attempted to deter- 
mine which standard and nonstandard features of the perused books were 
remembered best. The subjects were not told the purpose of the experi- 
ment, so that what they remembered about the books was presumably fairly 
close to what they would naturally have remembered in a real-life browsing 
situation. The "library" from which they were allowed to make their 
selection consisted of a previously assembled collection of approximately 
180 books in the subject area of psychology. No attempt was made 
this experimental collection a random sample of all existing books on 
psychology; instead, an attempt was made to cover a variety of topics in 
psychology which were thought to be of interest to the lay reader, as well 
as to Include a wide range of nonstandard features. In all. 104 experi- 
mental subjects were tested on up to five books each, and a total of 440 
usable questionnaires were obtained. 

In addition to standard author-title-subject data, the question- 
naire interrogated the subjects about the following nonstandard features 
of the books they had perused : 

1. Color of cover (whether white, gray, orange, etc.) 

2. Binding (spiral, cloth, etc.) 

3. Approximate height (samples supplied) 



4. Approximate number of pages 

5. Condition (new, somewhat used, etc.) 

6. Approximate date of publication 

7. Type of work (handbook, textbook, etc.) 

8. Level of work (popular, professional, etc.) 

The questionnaire also Inquired whether or not the book in question 
possessed the following features. These will later be referred to as 
"binary" features because only two answers (feature present and feature 
absent) are appropriate to such questions. 



9. 



Index 



10. Tables 

11. Figures, charts, diagrams 

12. Chapter titles 

13. Quotations from literary works 

14. Case studies 

15. Graphs 

16. Prefatory material 



17. Footnotes 

18. Bibliography’ or suggested readings 

19. Dedication 



20. Glossary 

21. Problems at ends of chapters 

22. Single volume (i.e.. Is the book a single volume work?) 

23. Translation (i.e.. Is the book translated?) 

24. Reprint or revision (i.e., Is. the book a reprint or 

revised edition?) 



This list of twenty-four features is thought to cover most, though not 
all, of the more obvious kinds of nonstandard information which come to 
mind as being of possible use. A few other kinds were touched upon in 
the questionnaire, but they will not be taken up here. 



To give a clearer impression of the nature of the questionnaire, 
the full text of the question on cover color is quoted below as an 
example of a typical question format: 

Color (Check the choice that most accurately describes the book.) 

blue or blue green ^black, dark gray, green 

dark blue, etc. 



brown or tan 

orange 

^ray 



red 



purple or violet 
yellow 



white 
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The ten color categories listed in this question were settled upon more 
or less arbitrarily by the experimenters. A sheet of special instruc- 
tions was issued along with the questionnaire one of which informed the 
subjects that it was permissable to check two or more color categories 
if necessary to describe a multicolored book. Most of the questions 
followed a format similar to the above one, except that some-to be 
specific, the questions on size, date of publication, and number of 
pages-did not have the format of a multiple choice question but instead 
provided a single blank for an approximate figure. Note that no 'don t 
know" blank was provided as part of the question on cover color. The 
absence of specially provided "don't know" blanks may have prompted 
some subjects to guess at many of the questions as best they could, a 
possibility whose implications will be discussed presently. 

The pertinent data obtained from the memory experiment is tabu- 
lated in Appendix D. It consists of twenty-four tables numbered 
la through 24a, with one table corresponding to each nonstandard feature 
type listed above. Each table constitutes a cross classification of 
questionnaires by correct answer and by response actually given. Data 
from single blank questions such as the one asking the approximate date 
of publication were put into this matrix form by choosing a series of 
discrete ranges and interpreting the response as thought it had been a 

choice from among such ranges. 

4, Preliminary Overview of E xperimental Result s 

In examining the data compiled from the memory experiment, the 
two most Obvious issues which come to mind are the following. First, 
what was the proportion of subjects who attempted an answer, as opposed 
to leaving the question blankl And secondly, of those attempting an 
answer, what percentage was correct! These two proportions can be 
readily determined for any nonstandard feature of interest using the 
tables in the appendix. The proportions of subjects attempting to 
answer ranged from a low of 29 per cent for the question on whether the 
book contained a dedication, up to 99 per cent for the question 
level of readership for which the work was intended. The median response 
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rate for all of twenty- four features was 74 per cent. The proportions 
of attempted answers (other than "don't know") which proved to be correct 
ranged from 24 per cent for the inquiry regarding the approximate number 
of pages up to 97 per cent for the question of whether the book was a 
single volume work, with a median accuracy rate for all features of 
72 per cent. In order to obtain the accuracy rates some rather arbi- 
trary decisions had to be made in some cases as to what constituted a 
"correct" response. For this and other reasons, it would be unwise to 
attach much significance to the exact figures just given. They are in- 
tended to convey a general impression only, to wit, that the response 
rate was on the whole rather high but that the accuracy rate among those 
who responded was for the most part unimpressive. 

The high response rate and low accuracy rate observed for many 
of the questions suggests the possibility that the subjects may have been 
indulging in a good deal of guesswork. Because a "don't know" blank 
was not provided for all questions, and because there was no specific 
directive asking the subjects to indicate "don't know" responses by leav- 
ing questions blank, it is possible that there was a tendency for the 
subjects to make a stab at all questions which had a multiple choice 
format, whether they really knew the answer or not. Such a tendency 
would hardly be surprising in subjects who were students accustomed to 
multiple choice tests of the if-you-don' t-know-make-a-guess type. There 
is no way of knowing just how strong the compulsion to guess was, although 
some evidence that it was not overwhelming is seen in the fact that almost 
all subjects left at least a few multiple choice format questions blank. 
The likeliest presumption is that the subjects did not do very much 
purely random guessing but that they nevertheless indulged in quite a 
bit of what might be called "semi-guesswork." That is to say, they prob- 
ably attempted to give answers to many questions on the basis of exceed 
ingly dim recollections or vague hunches. 

t 

If indeed a lot of semi -guesswork took place during the experi- 
ment, it was not necessarily a bad thing. In fact, it can be argued^ 
that a method of questioning which encouraged semi -guesswork was entirely 
appropriate for an experiment whose purpose was to gauge the usefulness 
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of nonstandard information for retrieval purposes. Briefly, the argument 
runs like this: When a retrieval system is unable to retrieve just one 
document for the patron with certainty that it is the one he wants, the 
next best thing it could do is, in effect, to determine for him an 
optimal order in which to search the collection. That is, it would be 
the task of the retrieval system to retrieve documents for the user's 
inspection in such an order that documents with the highest probability 
of being the wanted one would be presented to the user first. Even a 
vaguely recalled and possibly mistaken feature described by the patron, 
if exploited in the right way, could be used to improve this search 
order somewhat; for the feature in question presumably has at least a 
slightly greater probability of being as the patron describes it than 
it has of being any other way. Semi-guesswork should therefore actually 
be encouraged at future automated catalogs, and so it is appropriate 
that nothing was done to discourage semi -guesswork in the memory experi“ 
ment. However, care is required that the data from the experiment be 
analyzed in such a way as to take the possibility of guesswork properly 
into account. 

5. The Memorability of Nonstandard Information 

The most obvious factor of importance affecting the potential 
usefulness of a given type of nonstandard information is its memorability. 
Unfortunately, the "memorability" of a feature is not an easy thing to 
define. The most obvious suggestions are quickly seen to be inadequate. 
For example, one might consider measuring the memorability of a feaimre 
by the percentage of questionnaires in which the question about that 
feature was answered correctly; but unfortunately this percentage fails 
to indicate whether the remaining questionnaires had wrong answers or 
no answers at all, a matter which is clearly significant in judging 
memorability. 

The basic problem in using the experimental data to determine 
"memorability" is the unknown amount of guesswork and semi -guesswork 
which went on in the experiment. This means that any given correct 
answer may have been merely a lucky guess; it is certainly no proof in 
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and of itself that the subject really remembered anything. To meet this 
problem, it is essential that the data be analyzed in such a way as to 
evaluate the subjects' performance through comparison with what their 
performance would have been if they had adopted a strategy of guessing 
at all times. That is, we must ask; How much better did the subjects 
do by availing themselves of their recollections than they could have 
been expected to do through random guesswork alone? 

For concreteness let us examine the figures found in Table 9a 
of Appendix D. A reasonable approach would be to- compare these figures 
in some sensible way against what they might have been if the subjects 
had used a chance device to determine their answers. The first such 
device to come to mind is a fair coin. Are the figures in Table 9a con- 
sistent with the hypothesis that in all cases where a definite response 
was given, the subject had merely flipped a coin, answering that the 
book had an index if it came up heads and that it did not if it came 
up tails? Under the fair coin flipping hypothesis, one would expect 
the numbers of "index present" answers and "index absent" answers to 
be approximately equal, but the results show that the former outnumbered 
the latter by 212 to 87. Such a disproportion is extremely unlikely 
under the fair coin hypothesis and so that hypothesis must be rejected. 

The assumption that the random device used was a fair coin was 
purely arbitrary, and an observation which suggests a more enlightening 
test. Could the subjects have been using a biased coin, one weighted 
in such a way that the expected ratio of heads to tails was 212 to 87? 
Testing the biased -coin hypothesis is equivalent to testing the hypo- 
thesis that the subjects were merely guessing, but doing so with a marked 
tendency to guess "index present" rather than "index absent," the odds 
being 212 to 87 in favor of the former. Such a tendency could easily 
appear without the memory of the book in question being its cause j for 
example, a tendency to guess "index present" could be occasioned by the 
general knowledge that most books have indexes. Hence the necessity to 
find out whether the obtained responses were correct more often than 
would be expected from such guesswork. Now if a coin biased to produce 
on the average 212 heads for every 87 tails had been used, one would 
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expect that for books with indexes the number of "index present" answers 
and "index absent" answers would be approximately the ratio of 212 to 87, 
and the same for books without indexes. On this basis, the cell entries 
to be expected under the biased coin hypothesis can be calculated. The 
calculated values are shown in parentheses in the upper four cells of 
Table 9a (2). The observed numbers in the table are not strikingly 
different from those in parentheses, indicating that as a group the 
responding subjects did only slightly better on this nonstandard feature 
by applying their memories than they could have been expected to do with 
an appropriate chance device. 



TABLE 9a (2) 



data from memory experiment concerning indexes 

AUGMENTED BY EXPECTED VALUES 



Remembered 


Actual Property 




Property 


Index 

Present 


Index 

Absent 


Total 


Index present 


155 

(145.3) 


57 

(66.7) 


212 


Index absent 


50 

(59.7) 


37 

(27.3) 


87 


No response 


96 


45 


141 


Total 


301 


139 


440 



In cases where the difference between computed and observed re- 
sults is small, it is customary to apply a statistical test to see whether 
the difference might reasonably be accounted for by chance factors alone. 
An appropriate test (the two-tailed Chi-square test at the .05 signif- 
icance level with Yates correction for continuity) was applied to the 
above table, and it indicated that the differences between the observed 
and computed values could not reasonably be attributed wholly to chance 
factors. Stated more precisely, results as different from expectations 
as those observed would have less than one chance in twenty of occurring 
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under the biased coin hypothesis, and so the hypothesis may be rejected. 
This is statistical assurance that at least some of the subjects must 
have been using their memories rather than their guessing facilities 
on this question. The same test was made for the remaining six-cell 
tables in Appendix D with similar results in all cases but one. 

That case was the question on whether the book had problems at the ends 
of its chapters, the responses to which turned out not to be significantly 
more accurate than random guessing. 

Comparing the four parenthesized figures in the above table with 
their unparenthesized counterparts gives some idea of the memorability 
of the index feature of books. However, it would be desirable to sum- 
marize the memorability of indexes in a single figure. As a step in 
this direction, it is possible to condense the six-cell table down to 
four cells by combining the "no response" row with the two upper rows. 

This is done as follows: One assumes that the 141 subjects giving no 

response had no recollection of the feature whatsoever. Therefore, if 
they had been forced to make a response, that response would have been 
a random guess. Suppose that a certain proportion p of the 141 had 
guessed "index present," and the remaining fraction (1-p) had guessed 
"index absent." The results would have been as shown in Table 9a (3). 

This table shows the quantities from the upper four cells of Table 9a (2) 
augmented by the expected numbers of additional guesses which would have 
accrued to these cells if the no response subjects had been forced to 
make wild guesses. In effect, collapsing Table 9a (2) into Table 9a (3) 
changes the rules of the experiment after the fact in such a way as to 
remove the complicating factor of null responses. 

A single figure indicating memorability can be obtained from 
Table 9a (3) simply by subtracting the expected percentage of correct 
answers from the observed percentage of correct answers. The observed 
number of correct answers is 155 + 96p +37+45 (1-p) and the number 
expected if all subjects had guessed is 145.3 + 96p + 27.3 + 45 (1-p). 
Converting to percentages and subtracting, one obtains a difference of 
4.4 per cent. This figure, which represents the percentage of correct 
answers over and above the percentage to be expected from guesswork, we 
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shall call the "memorability level" of the feature. It is possible to 
compute the memorability level in a similar way for all the nonstandard 
features under consideration, including those whose tables in the 
appendix consist of more than six cells. 



TABLE 9a (3) 

DATA FROM MEMORY EXPERIMENT COLLAPSED INTO FOUR CELLS 
AND AUGMENTED BY EXPECTED VALUES 



Remembered 


Actual Property 


Total 


Property 


Index Present 


Index Absent 


Index present . 


155 + 96p 
(145.3 + 96p) 


57 + 45p 
(66.7 + 45p) 


212 + I4lp 


Index absent 


50 + 96 (1-p) 
[59.7 + 96 (1-p)] 


37 + 45 (1-p) 
[(27.3 + 45 (1-p)] 


87 + 141 (1-p) 


Total . . 


301 


139 


440 



In Table 1,'the twenty-four nonstandard features are ranked 
according to their memorability indexes. From the table, it is seen 
that the approximate number of pages was the most memorable feature of 
a book to the subjects of this experiment. The color of the cover, which 
one might have supposed would be among the most memorable features, ranks 
only ninth. The least memorable aspect of a work among all those investi- 
gated was whether it had problems at the ends of its chapters, which is 
not surprising in view of the previously mentioned finding that the 
observed responses did not even show a statistically significant dif- 
ference from random responses for this feature. 

The average feature showed an improvement of only 9 per cent in 
number of correct answers over what would be expected from sheer guesswork, 
and the most memorable feature an improvement of only 19 per cent. This 
is hardly an impressive sliowing. However, in fairness to the mental 
abilities of the subjects, it must be added that not all mistaken responses 
were necessarily occasioned by faulty memory. Some of what were recorded 
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TABLE 1 



NONSTANDARD FEATURES RANKED IN ORDER OF DECREASING MEMORABILITY 



Nonstandard Feature 


Number of 
Response 
Categories 


Memorability 

Level 




6 


19% 




2 


18 




6 


17 




2 


16 




10 


15 




13 


15 




2 


13 




7 


12 




11 


11 


Quotations from literary works 


2 


10 




8 


9 




5 


9 




2 


8 




2 


7 




2 


7 




2 


5 




2 


4 




2 


4 




2 


4 




2 


3 




2 


3 




2 


2 




2 


1 


Problems at ends of chapters 


2 


0.4 




4 


9% 
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as mistakes may have been a matter of mere semantics --of terminological 
differences between experimenters and subjects. As a case in point, 
there was a strong tendency for subjects to classify as "black, dark 
gray, dark blue, etc." books which the experimenters had judged to be 
"blue or blue-green." The experimenters themselves commented on the 
difficulty of deciding in which of these two classes to put certain 
moderately dark blue books, so it would hardly be surprising if there 
had been some differences of judgment on this point between experimenters 
and subjects. There are indications that where type of work and condi- 
tion of work were concerned, rather wide differences in judgment arose 
between the experimenters, who were accustomed to the terminology of 
librarianship , and the subjects, who were not. The extent to which 
such differences of judgment contributed to the total number of osten- 
sibly "incorrect" answers obtained in the experiment is not known. 
Whatever the extent, however, any semantic difficulties affecting the 
experiment could well affect the success of future retrieval systems 
also, and so for present purposes they can be viewed as influences 
tending to make the experiment more realistic. 

The measure of memorability level employed in Table 1 has many 
good aspects. For example, because it involves a comparison with random 
guessing, it is not greatly affected by the arbitrary decision as to 
how close a subject's recollection of the number of pages in a book must 
be to the actual number of pages in order that his estimate be considered 
"correct." On the other hand, it is necessary to bear in mind that this 
measure of memorability has limitations. Its most serious defect is 
that it has a tendency to favor features with many response categories 
over those with few; at least, this is so under certain plausible inter- 
pretations of the word "memorable." In consequence, it is not clear 
that, for example, a book's color is really more memorable than its 
footnotes, for the better showing of the former in terms of the memor- 
ability level may be due mainly to the color question's larger number 
of allowable responses on the questionnaire (eleven as opposed to two). 
However, features with the same or nearly the same number of answer cate- 
gories can be compared with reasonably safety. The number of answer 
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categories for each feature is included in Table 1 to aid in its inter- 
pretation. Another limitation of the measure is that it makes no 
allowance for degrees of mistakenness. For example, an estimate of 
book's length which is in error by only 100 pages is treated as if it 
were just as mistaken as an estimate which is out by 500. 

Conceivably some more elaborate measure of memorability could 
be constructed to overcome the dependence of the "memorability level" 
measure on the number of response categories. However, there is some 
question whether one could ever discover a completely convincing 
mathematical formulation for such a vague concept as "memorability." 

In any case, it would seem that efforts to construct a more refined 
measure would be better directed toward explicating the concept of 
"retrieval usefulness" instead of "memorability." It is the utility 
of a book property for retrieval purposes which is of ultimate interest 
and it will shortly be seen that memorability is only one factor affect- 

ing retrieval usefulness. 

6. V,.,^e.-red Sea r ch T.eneth as a Measure nf Retrieval Usefulnes s 

How can "retrieval usefulness" be gauged? The measure which 
will be employed here is based on the concept of ex pected search leng th 
and its companion notion, the reduction fact^ in expected search lengt . 
These concepts are presented in detail elsewhere, [5] in the more gen- 
eral setting of subject retrieval. The basic notions are equally 
applicable to specific work retrieval however, and in this context they 
are simple enough to be conveyed by a few examples. 

By the "expected search length" of a request is meant the number 
of unwanted documents which the requester could expect to have to search 
through randomly before finding the wanted one. For example, if the 
wanted document is known to be In a set of 201 documents, the user 
could examine the documents in the set one at a time in random order 
until he comes upon and recognizes the one he is affer. If he does 
this, the wanted document might turn up early in his search and it 
might turn up late, but the statistical expectation is that this user 
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will have to look through exactly 100 unwanted documents (i.e., half of 
the unwanted documents in the set) before coming to the wanted one. 

A nontechnical interpretation of this statistical expectation is that 
if he were to repeat his random search for many hypothetical trials, 
the average number of unwanted documents he would be forced to discard 
in each trial would be approximately 100. 

This simple idea contains the germ of a method for the evaluation 
of nonstandard clues. For the sake of a clear example, let us make some 
simplifying assumptions. First, let us assume that we wish to evaluate 
a nonstandard clue type’s potential not in combination with other non- 
standard information, as it would almost certainly be used in practice, 
but instead in complete isolation. That is, we imagine a hypothetical 
library in which a certain type of nonstandard clue is the only, type of 
clue, standard or nonstandard, which a requester is allowed to submit. 
The question of interest is how much the library's retrieval system 
would be able to do for the requester with this one clue. Secondly, let 
us assume for simplicity that the clue type in question is of the binary 
sort. Thirdly, let us put aside for the moment the problem of poor 
memory and assume that every requester in this library can be counted 
on to remember, and remember accurately, whether or not the book he 
wants has this binary property. Finally, let us make up some statistics 
about this hypothetical library: (1) The size of the collection is 

only 201 documents; (2) 61 of the documents have the property and the 
remaining 140 do not; (3) when users submit requests to this library, 

40 per cent of the time it is a document having the property which is 
wanted and 60 per cent of the time it is not. 



Obviously no retrieval system could isolate the wanted book on 
;he basis of this single binary clue, but it could isolate the ^ of 
111 books which answer to the clue, and this will decrease the requester’s 
expected search length. Suppose a requester submits a request consisting 
of the answer "feature present’’ to the question of whether the book he 
wants has the property. The set of 61 documents having the property 
could be retrieved for his inspection on the basis of this clue. One of 
the 61 is the book he wants, so if he searches this retrieved set randomly. 
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his expected search length will be one-half of 60 or 30 unwanted docu- 
ments. Similarly, the expected search length for a "feature absent" 
request is half of 139 or 69.5 unwanted documents. With 40 per cent of 
the incoming requests resulting in an expected search length of 30 and 
60 per cent in 69.5, the average expected search length will be .4(30) 

+ .6(69.5) == 53.7. This means that the number of unwanted documents 
which would have to be searched through should average approximately 
54 per request in the long run in this library. 

The figure of 54 can be put in perspective by comparing it with 
what the average expected search length would have been if not even 
the single binary clue had been utilized. As previously calculated, a 
totally random search would have an expected search length of 100. 
Exploiting the clue reduced this by 100 - 54 = 46; that is, the ex- 
pected search length was reduced by a factor of 46 per cent as against 
what it would have been if the retrieval process had made no use of the 
clue. The figure of 46 per cent we call the expected search lengtji 
reduction factor associated with the binary property. It can be inter- 
preted to mean that using the binary property as a clue for retrieval 
would save the average user 46 per cent of the search effort he would 
otherwise have had to expend to search out his book randomly. Speaking 
more generally, it can be viewed as a measure of the potential utility 
of the nonstandard feature in question. 

Some of the simplifying assumptions made for the sake of the 
above example are inessential. The calculation of the expected search 
length reduction factor for nonbinary features is a direct extension of 
the binary case. The assumption of a definite collection size is not 
really needed, since the reduction factor is virtually independent of 
collection size except for very small collections. Hence statistic 
(1) could have been dispensed with and (2) restated in terms of per- 
centages. In section 8, it will be shown how the possibility of mis- 
taken information can also be taken into account. 
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7. The Role of Collection Statistics 

Although the exact relationship may not be immediately apparent, 
it seems clear that the relative abundance or paucity of books having 
a certain property must have something to do with how useful that prop- 
erty would be as a retrieval clue. To take a colorful example, there 
is reputed to be a rare edition of a book on reptiles, the binding of 
which contains a rattlesnake's rattle which is activated whenever the 
book is opened. Now the property of containing such a rattle is pre- 
sumably a highly memorable property. However, it is such a rare property 
that it would hardly be worthwhile to ask the average library patron 
whether his book rattles when opened, because the chances are so remote 
that it does. Evidently then the memorableness of the property is not 
the only consideration; the "collection statistics" also affect utility. 

The example in the last section showed that the collection sta- 
tistics which enter the question are of two distinct types. One is what 
could be called holdings statistics , and pertains to how the documents 
on the library shelves are distributed among the possible categories of 
a feature. The other type may be called usage statisti cs and pertains 
to the distribution of a feature's categories observed among incoming 
specific work requests. To give insight into how collection statistics 
affect retrieval utility, let us once again consider the hypothetical 
example of a binary feature which is always remembered accurately. 
Assuming for the moment that there is no noticeable difference between 
the holdings and usage statistics for a given property, it is possible 
to compute the expected search length reduction factor for any statistics 
of interest by the method of the last section. For example, if 30 per 
cent of the documents held in a collection have the property and likewise 
for documents requested, the expected search length reduction factor 
turns out to be .42. (Actually the figure is only approximately .42, 
although it will approximate it more and more closely as larger and 
larger collection sizes are assumed.) By generalizing this calculation, 
the reduction factor associated with any given proportion x as the 
assumed collection statistic can be shown to be 2x(l-x). 
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The relationships expressed by this formula is displayed graph- 
ically in Figure 1. Observe that the utility of a binary feature as 
measured by the expected search length reduction ratio vanishes as x 
approaches 0 or 1. This confirms what was previously brought out on 
intuitive grounds alone, namely that a rare property is of negligible 
usefulness no matter how memorable. Observe also that the feature is 
of maximum utility if x = .5-that is, if the property "divides" the 
collection evenly. Finally, it is of interest that the reduction ratio 
is not overly sensitive to the collection statistics so long as x is 
in the general vicinity of .5. Indeed, the proportion can vary between 
.3 and .7 without causing the reduction factor to leave the narrow 
range between .42 and .5. Although it will not be demonstrated here, 
the same sort of general observations can be extended to nonbinary 
properties and properties which are not totally memorable. 




Figure 1. Expected search length reduction factor as a function of the 
proportion of the collection having a. binary property of 
interest. Perfect memorability is assumed. 
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The dependence of potential usefulness on collection statistics 
raises a problem. It is impossible to gauge a feature's potential use- 
fulness for retrieval, either absolutely or relative to that of other 
features, without making assumptions about the collection for which its 
use in retrieval is contemplated. What assumptions should be made? No 
library collection is exactly like any other, so if the nonstandard 
features are evaluated relative to one particular library, the evalua- 
tions obtained will not be entirely valid for any other library. The 
best that can be done here is to make the evaluations with respect to 
some more or less typical library collection, so that at least a gross 
impression can be gained of what their value would be in other collections. 
Then, if a reader has special need of a more precise evaluation relative 
to some other collection, it should be possible for him to repeat our 
calculations using the statistics of that particular collection. 

The collection statistics which we shall adopt here are estimates 
obtained from two random samples of the collection of the main library 
at the University of Chicago. The first sample, which provided the 
holdings statistics, consisted of 125 documents drawn with the help of 
random number tables from the main holdings list. The second, which 
yielded usage statistics consisted of 106 documents drawn randomly from 
those returned to the circulation desk by the library patrons. These 
seemingly small samples were in fact sufficiently large so that all 
estimates of proportions could be made with .95 confidence intervals 
whose widths ranged between +3 per cent and +9 per cent. This was felt 
to be sufficient accuracy in view of the fact that the expected search 
length measure of utility for which the data is required is less sensitive 
to collection statistics than to statistics pertaining to memorability. 
There is also an argument against the need for large samples arising 
from circumstance just discussed, namely, that the statistics of other 
library collections are not necessarily identical with those of the 
University of Chicago library anyway. 

The raw data obtained from the samples is tabulated in Tables lb 
through 24b of Appendix D. As one would expect, some features showed 
collection statistics which were much more favorable for retrieval purposes 
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than others. Among the binary features, the following seven were dis- 
tributed in the library holdings in the most favorable way: index, 

bibliography, footnotes, tables, figures, preface, and quotes j each of 
these occupied a proportion of the library between .3 and .7. Three 
features - -glos sary , problems, and single- volume works --showed extremely 
unfavorable distributions, lying outside the range .04 to .96. There 
were no truly striking differences found between the holdings statistics 
and the corresponding usage statistics, although for some features there 
was a difference large enough to be statistically significant (at the .05 
significance using the Chi-square test) in spite of the smallish sample 
sizes. These included the type of work, level of work, type of bxnding, 
condition, chapter titles, index, and presence of literary quotations. 

8. Expected Search Length for a Possibly Mistaken Request Clue 

The hypothetical one -clue -library method of evaluation illustrated 
in Section 6 is easily extended to apply to nonbinary features. We shall 
show next how it can be generalized to apply as well to information 
which may be mistaken. Consider a patron who submits to the retrieval 
system the information that the book he wants is red. Because his memory 
may be faulty, there is no subset of the collection which can be said 
with certainty to contain the book he is really after. By the same 
token, no book or set of books can be eliminated from consideration en- 
tirely. The most that can be said is that the books in a certain set-- 
probably but not necessarily the set of all red books— have a higher 
probability of being what is wanted than do any of the remaining books. 
Thus all inferences drawn from the patron’s stated clue must be prob- 
abilistic in nature; they must be inferences to the effect that certain 
books are more likely to be the one wanted than are others. 

Suppose now that a retrieval system uses an appropriately pro- 
grammed computer or is in some other way able actually to compute for 
each book in the collection the probability that it is in reality the 
wanted one, given that the requester believes his book to be red. The 
highest probability figure will be assigned to the members of a certain 
color class, presumably the class of red books. The next highest 
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probability will be assigned to members of some other class--the orange 
books, perhaps, if a strong tendency has been observed among requesters 
to confuse red and orange in their memories. The third highest prob- 
ability will go to each member of still another class, and so on until 
each of the eleven color classes has been assigned a number which is 
the probability that any given member book is the required one. The 
obvious retrieval strategy is the following: retrieve for the requester's 

examination the books with highest probability first. If he fails to 
find his book by searching randomly through these but is willing to 
search further, retrieve next the books with second highest probability, 

and so on. 

The assumption involved here is essentially that the retrieval 
gygl;0ju ig able to respond to a requester's nonstandard clue by ranking 
all documents in the collection according to their probability of being 
wanted. This assumption will allow a calculation of the potential utility 
of that clue expressed in terms of an expected search length reduction 
factor. The calculation will show the maximum utility of the clue which 
\7ould be realizable in isolation from other clues, for it seems clear 
that to rank the documents by probability is the best that could be done 
for the requester on the basis of a possibly mistaken clue. The re- 
mainder of this section presents the details of how the probabilities 
can be calculated from the data in Appendix D, and how the expected 
search length reduction factor can in turn be computed from the prob- 
abilities. Readers willing to accept the mathematics on faith may wish 
to proceed directly to Section 9 . 

The first task is to deduce, for any given search request having 
the form of an isolated nonstandard book property, that ranking of the 
collection by probability which an ideal retrieval system would present 
to the requester as a response to his request. For the sake of con- 
creteness, let us continue to speak in terms of a color clue. Let the 
symbol "Black" denote the event that the request consists of the clue 
that the book has a black or very dark cover. That is, "Black" names 
the event that the library patron thinks the book he wants is black, 
and that he has so stated, say by checking the appropriate blank on the 
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request card. Similarly, let "Blue" denote the event that he thinks 
his book to be blue, and so forth through the eleven color categories. 

The symbol "No response" can be added to denote the event that the re- 
quester does not remember the color at all and so leaves the request 
form blank. Further, let "BLACK" denote the event that the book which 
our requester wants is really black, let "BLUE" denote that it is 
actually blue, and so forth for the remaining color categories. As a 
preliminary step toward deducing the wanted ranking, we calculate con- 
ditional probabilities of the form typified by pr (BLUE/Black) , which 
is the probability that our patron is really after a blue book, given 
that he says the book he wants is black. 

The calculation proceeds using Bayes' theorem 

pr (Black/BLUE) pr (BLUE) 

• pr (BLUE/Black) = 

pr(Black/BLUE)pr(BLUE) + ... + pr (Black/BLACK)pr(BLACK) 

The right side of the identity involves conditional probabilities such 
as pr(Black/BLUE) . By Table la of the appendix, there are ninety cases 
in which subjects were questioned about books which were really blue, and 
jLn twenty out of these ninety the subjects mistakenly remembered their 
book as being black, so the experiment indicated that pr (Black/BLUE) 
is approximately 20/90. A slightly better estimate of the true prob- 
ability is obtained by adding one to the numerator and the number of 
possible responses (including no response) to the denominator, yielding 
(20+1)/ (90 + 12) = .206 as the desired probability. (This correction is 
a generalization of the formula of Laplace [6, pp. 133-135]. Although 
controversial, it is nevertheless preferable to the uncorrected ratio 
which would lead to such absurd estimates as a probability of zero for 
small samples where the numerator happens to be zero.) The remaining 
eleven conditional probabilities on the right side are calculated simi- 
larly. The right side involves also quantities such as pr(BLUE), the 
probability prior to receiving his request that the patron* a desired 
book is blue. According to Table lb of the appendix, twenty-eight of 
the random sample of 106 books withdrawn from the University of Chicago 
library were blue. Making a Laplacian correction as before, we 'take 
(28+1)/ (106+11) = .248 as the estimate of the proportion of users who 
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desire blue books. The other ten colors are treated analogously. Upon 
substituting all these estimates into the Bayesian equation, one obtains 
the estimate pr(BLUE/Black) = .330. This figure can be interpreted as 
meaning that when a library patron states that the book he wishes 
locate is black, the chances are about one in three that it is really 
blue. Such an interpretation assumes of course that the patron is en- 
couraged to use guesswork to the same extent as were the experimental 
subjects, and also that the semantic difficulties associated with the 
color category labels are as they were in the experiment. The remaining 
ten conditional probabilities pr(RED/Black) , pr(BUCK/Black), etc., are 
calculated in a similar way. 

It may be asked why a probability such as pr(BLUB/Black) cannot 
be obtained directly from Table la of the appendix, instead of indirectly 
via Bayes' identity. The reason is that the book collection used in the 
memory experiment was not a random sample of any real library collection, 
and the estimate of pr (BLUE /Black) so obtained would be contaminated by 
any accidental bias toward or away from blue books in experimental col- 
lections. In contrast, the conditional probability pr(Black/BLUE) can 
be safely obtained from Table la and is not affected by any such bias 
the experimental collection might have had. It might also be asked 
whether the use of Bayes' theorem is not objectionable. In spite of 
the controversy surrounding Bayes' theorem and the notorious misappli- 
cations of it which have been perpetrated in the past, its application 
in the present situation is felt to be legitimate, especially since the 
prior probabilities used are not mysterious assumptions but empirically 

obtained estimates. 

With the foregoing calculations completed, the assignment of 
probabilities to individual documents is easy. We wish to assign to 
each document in the entire collection its probability of being the one 
truly wanted. Suppose the collection contains a total of H documents. 
Then the number of, say, blue documents it can be assumed to contain is 
obtained from the holdings sample findings of Table lb in the appendix. 
According to the table, 26 of the 125 books sampled were blue, which 
leads to the assumption that the number of blue books in the hypothetical 
library equals the prop.ortion (26 + 1)/(12S + 11) times N, or .199 N. 
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We have already determined that when a user thinks his book is black, 
the probability of its really being blue is .330. In the absence of 
information to the contrary, each blue book can be considered equally 
likely to be the one in question. It follows that each blue book has 
probability .330/.199N = 1.66/N of being the wanted book. Treating all 
eleven colors in this way, one can determine the ideal ranking in response 
to the request "Black." It has eleven ranks or "levels" in the following 



order : 

.081 N black books each having probability 

.199 N blue books each having probability 

.029 N gray books each having probability 

. 103 N brown books each having probability 

.007 N purple books each having probability 

.015 N yellow books each having probability 

.015 N orange books each having probability 

. 154 N multicolored books each having 

probability 

.007 N white books each having’ probability 

, 118 N green books each having probability 

.272 N red books each having probability 

This list describes the best possible ranking with which a retrieval system 
could respond to the isolated request clue "Black." To illustrate the 
meaning of the list, in a 1,000 document collection constituted like the 
University of Chicago library, there would be about eighty-one black 
books, and these, having a relatively high probability of about .002 
each, should be retrieved first. If the patron does not find his book 
among these but still wishes to continue his search, the 199 blue books 
should be presented next; and so forth. The rankings which would be 
required by some other clue that a requester might give, such as his 
statement that his book is blue, or even his statement that he does not 
remember its color, can be constructed in a completely analogous fashion. 

Let us turn now to the problem of evaluating the average use- 
fulness of these rankings. A formula for the expected search length of 
a request in any arbitrary collection ranking is derived in [5]. When 
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specialized to the case of specific work retrieval, it becomes 

1 

esl(q) = j + 2 

where q is the query or search request in question, j is the total 
number of documents in all levels of the ranking which precede the 
level containing the wanted document, and i is the number of unwanted 
documents in the level containing the wanted document. This formula 
merely states that a requester determined to find his book would have 
to search through all the books in all the levels preceding the level 
in which he will eventually find his wanted book, plus an expected 
total of half the unwanted books in that level. For example, the ex- 
pected search length for a user who submits the request clue Black 

when the book he wants is really blue would be 

. 199N - 1 

esl (Black) = .081 N + . 

BLUej 2 

If N is assumed to be large this expression can be simplified by deleting 
the minus one. 



The special case where the wanted book is really blue is of no 
particular interest in itself, but the value of eslg^^y^, (Black) can be 

combined with similarly computed values for esl^^^ (Black), esl^^^^ 
etc., using the formula: 

esl(Black) = pr (BLUE/Black) esl^^^yg (Black) + ... + 

pr(BLACK/Black) eslg^A^j^ (Black). 

This formula weights each special case's expected search length by the 
probability that that special case will arise from the request "Black. 

The resulting value is an expected search length free of special assump- 
tions about where the wanted document really is in the weak ordering. 
Substituting previously obtained figures for the probabilities one obtains 
esl(Black) = .353N. To illustrate the meaning of this figure, in a 
library of 1,000 documents, a user submitting the lone clue "Black" 
could expect to have to search through 353 documents, on the average, 
before finding the wanted one. 
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Even the value of esl(Black) is of little interest in and of 
itself, since "Black" is merely one of twelve possible responses to inter- 
rogation about color. What will the expected search length be on tj^ 
average after a patron has been asked the color of the book he desires? 

This average can be obtained by weighting the expected search lengths 
for particular colors by the probabilities of those colors being given 

as clues. In symbols, 

iil = pr (Black) esl(Black) + ... + pr(Blue) esl(Blue) 

+ pr (No response) esl(No response) 

where e71 denotes the expected search length averaged over all users of 
the system. The probabilities in the right side can be expanded using 

identities of the form 

pr (Black) = pr (Black/BLUE)pr (BLUE) + ... + pr (Black/ BLACK) pr (BLACK) . 

The conditional probabilities in this latter expression have already been 
estimated from Table la of the appendix, and its remaining probabilities 
we have already estimated as usage statistics from Table lb of the 
appendix. Substitution of these values yields the estimate pr(Black) 

,155, and the other probabilities are similarly obtainable. With all 
the necessary ingredients now available, esl can be computed and turns 
out to be .353N. 

We conclude that by exploiting nonstandard cover color information 
it 'would be possible to attain an expected search length of around .353N 
documents for the average user of a library of N books. An average ex- 
pected search length of . 5N is achievable by totally random searching. 

The expected search length reduction factor is therefore (.5N - .353N)/.5N, 
or .294. In other words, the use of a patron's color memory could be 
expected to cut down the length of his search by about 29 per cent on 
the average. This percentage can of course be computed for any of the 
nonstandard features, not just cover color, and provides a convenient 
basis for comparing the retrieval utility to be expected from one type 
of feature with that of others. 
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9. The Relative Usefulness of the Various Clue Types in Isolation 

The first two columns of Table 2 constitute a list of all twenty- 
four clue types together with their corresponding expected search length 
reduction factors in descending order of the latter. As previously 
explained, a reduction f .tor represents the percentage by which a 
library user's search effort for his book would (on the average) be cut 
down if he could be given an opportunity to state his recollection of 
the feature in question as an input request to an appropriate retrieval 
system. It is seen from the table that a properly utilized nonstandard 
clue could be expected to reduce the length of a patron's search by as 
much as 57 per cent or as little as 3 per cent, depending on the clue 
type. The reduction factors of the various clue types are spread fairly 
evenly between these extremes, and average 21 per cent. 

The first observation to be made about Table 2 is that none of 
the reduction factors are impressively large. One might have hoped, for 
example, that by exploiting a library patron’s knowledge of a book’s 
color it would be possible to reduce the amount of searching he would 
have to do to find the book by 80 or 90 per cent--that is, by a factor 
of five or ten. Instead, the experimental results indicate that not 
even a factor of two would be achieved. This makes it clearer than ever 
that no single clue type would be of much use in isolation. A second 
and related observation is that there are some clue types, such as the 
presence in a book of a dedication, a glossary, or problem sets, which 
show so little promise of retrieval usefulness that it is questionable 
whether it would be worthwhile to make use of them at all. Thirdly, 
there is a tendency in Table 2 as there was in Table 1 for clue types 
with many answer categories to appear near the top of the ranking, while 
those with few or only two answer categories appear lower down. Apart 
from this, it is difficult to discern any very striking pattern in the 
ranking. There is, for example, no noticeable tendency for the "intel- 
lectual" attributes of a book such as its level and type to rank either 
much higher or much lower than "physical" attributes Wxth a similar 
number of answer categories. 
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TABLE 2 



NONSTANDARD FEATURES RANKED IN ORDER OF DECREASING 

POTENTIAL USEFULNESS 



1. 

Nonstandard Feature 


2. 

Expected 

Search 

Length 

Reduction 

Factor 


3. 

Number 

of 

Response 

Categories 

1 


4. 

Expected Search 
Length Reduction 
Factor Per 
Stored Bit 

(Col. 2 i log^ Col. 3) 


Date 


577o 


10 


177o 


Type of work 


49 


13 


13 


Number of pages . . . 


43 


6 


17 


Binding 


32 


8 


11 


Color 


29 


11 


9 


Level 


27 


6 


11 


Height 


27 


7 


9 


Quotes 


26 


2 


26 
*1 *1 


Condition ...... 


26 


5 


11 


Index 


23 


2 


23 


Figures 


21 


2 


21. 


Tables 


21 


2 


21 


Chapter titles . <. . . 


20 


2 


20 


Graphs 


18 


2 


18 


Footnotes 


18 


2 


18 


Case studies 


16 


2 


16 


Translation 


15 


2 


15 


Preface 


14 


2 


14 


Bibliography 


8 


2 


8 


Single volume . . . . 


7 


2 


7 

n 


Reprint or revision . 


7 


2 


/ 

c 


Dedication 


5 


2 


D 


Glossary 


3 


2 


3 

o 


Problems 


3 


2 




Average 


21.47o 


4.0 


13.57o 



-114- 



The tendency for features with many response categories to appear 
high in the list has a straightforward explanation. When a patron is 
questioned about a many-category feature such as color, his reply con- 
tains more information (in the information-theoretic sense) than would 
his answer to a question about, say, a binary feature, such as the 
inclusion of an index. This is true because a multiple choice question 
offering many choices is by nature capable of eliciting more information 
than one offering only two choices. The same point can be made in terms 
of partitions of the document collection: A binary feature can only 

divide the collection into two subsets, whereas nonbinary features can 
partition it further and so permit a better focusing of the requester s 
search efforts. The ranking in Table 2 is "unfair" insofar as it fails 
to correct for the natural advantage enjoyed by the features with many 

response categories. 

These admittedly vague remarks can be made more concrete by 
considering the amount of computer storage space which would be taken up 
by the various types of document descriptions. A binary feature would 
require only a single binary digit of storage per document, with, say, 
a "1" to signify the presence of the feature in the document and a 0 
to indicate its absence. A four-category feature would require two bits, 
an eight-category property three, and so on, with each n-category feature 
requiring approximately Log 2 n bits of storage space. In an automated 
catalog where computer st^.age space was a crucial consideration, the 
utility of a feature would have to be considered in relation to its 
price, namely, the storage space needed to record it. In such circum- 
stances the critical quantity to be examined would be the expected search 
length reduction factor per stored bit , obtained by dividing the reduc- 
tion factor for a feature by the logarithm with base two of the number 
of categories subsumed by the feature. The storage problem is a real 
one, but even aside from any practical storage considerations, this 
quantity probably achieves as good a correction for the above mentioned 
"unfairness" as could be hoped for. In column 4 of Table 2, the expected 
search length reduction factors are given on a per-stored-bit basis. 
Column 4 should be consulted in lieu of column 2 if it is desired to 
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gauge potential usefulness relative to a given amount of stored catalog 
information rather than to a given number of questions to be answered 
by the library user. 

10. The Usefulness of Nonstandard Clues in Combination 

The assumption in force up until now that only a single non- 
standard clue would be used to effect retrieval was a highly artificial 
one. It was imposed only in order to obtain separate usefulness ratings 
for the different clue types. In actual practice, a large number of 
clues would have to be solicited from the library patron and used in 
combination in order for him to derive any substantial benefit from 

their use. 

Let us review in more detail how nonstandard clues could be used 
in combination. Suppose a library patron cannot remember enough standard 
information about a book to be able to locate it using conventional 
library resources, but can nevertheless remember some nonstandard infor- 
mation about it. In a future library equipped to cope with his problem, 
he might well be asked to fill out a search request card consisting of 
multiple-choice questions much like the twenty-four questions with which 
we have been concerned in this study. His completed request card would 
be submitted as input to a computer programmed especially to compute, 
for any combination of nonstandard properties, the probability that a 
book having that combination of properties is the book wanted by the 
requester. The computations would be based on statistics pertaining 
to the memorability of the properties--statistics obtainable either 
from memory experiments such as the one reported here or from the re- 
corded past experience of the system--and also on collection statistics. 
The results of the computations would be that books having such and 
such a combination of properties are the likeliest candidates, books 
with some other combination are next likeliest, and so on. The file of 
library holdings would then be searched automatically for documents 
belonging in the high- likelihood classes. The final output given to 
the user would be a bibliographic list of documents (or possibly the 
physical documents themselves) grouped in such a way that he can 
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conveniently scan the documents with the highest likelihoods first. 

If the computed probabilities are very low even for documents in the 
highest likelihood group (this would be the case, if, for example, the 
user had left most of the questions on the request card blank), he might 
wish to abandon his search at the start. Otherwise, he would scan the 
list until he had recognized the book he is after, although he might 
at any time give up either from lack of patience or lack of further 
documents having a sufficiently high probability to justify continued 
searching. No doubt there are many possible variations on this outlined^ 
procedure for exploiting nonstandard clues, but the essential element 
in all of them must be some means of obtaining for the user's inspection 
at least the first few levels of a ranking by probability of the document 

collection. 



The computations necessary to associate probabilities with prop- 
erty combinations are fairly involved. The complicating factor is that 
the nonstandard features of interest are in general not distributed in 
the document collection in such a way as to be statistically independent 
of one another. There is also no assurance of statistical independence 
between what a library user uends to remember concerning one type of 
feature and what he tends to remember about another. The generalization 
of the probability calculations of Section 8 in such a way as to take 
proper account of the troublesome dependencies, though not impossible, 
will not be carried out here. Instead, we shall merely present an 
extremely rough indication of how effective a retrieval system which 
could compute the proper probability rankings would be. 



One can readily obtain a crude estimate of the effectiveness of 
a given combination of clue types simply by ignoring the problem of 
statistical dependencies. To illustrate this, consider the case of a 
retrieval system which will accept both date and type of work clues. 

By Table 2, the use of information as to approximate date will (roughly 
speaking) cut down by 57 per cent the number of books to be searched, 
leaving 43 per cent. The type-of-work information will cut this down by 
a further 49 per cent leaving only 43 per cent - .49 (43 per cent) 

21 per cent of the original body of documents. The reduction factor for 



o 



-117“ 



the combination of these two leading clue types is therefore 100 per cent - 
21 per cent = 79 per cent. A first approximation to the reduction factor 
for any clue- type combination of interest may be obtained in the same way. 

Using this crude method of estimation, one finds that the expected 
search length reduction factor achievable from the simultaneous use of 
all twenty-four clue types investigated is approximately 99.8 per cent. 

This estimate is on the high side; it would probably have to be lowered 
substantially if the statistical dependencies were taken into account. 

On the other hand, if refinements were made in the use of the twenty- four 
clue types, and if in addition other clue types were utilized which are 
not among the twenty-four, the additional retrieval power so achieved 
would help to make up for this to some unknown extent. It is therefore 
not unreasonable to take the 99.8 per cent figure as it stands as about 
the best order-of -magnitude guess as to the combined potential retrieval 
usefulness of ''all" nonstandard types of information which can be made 
at this time. To the extent that the conditions of the memory experiment 
can be assumed to reflect future retrieval conditions, and to the extent 
that the statistics of the University of Chicago library are typical of 
document collections in general, it can be said that optimal utilization 
of all nonstandard information possessed by a library patron about a 
document with which he. has had previous contact would result in reducing 
by a ratio of something like 500 to one the. expected search effort 
necessary for him to locate the document in a large collection. 

Some tentative conclusions can now be drawn. An expected search 
length reduction factor of around 99.8 per cent would be very useful in 
a small library of, say, 10,000 documents, for if the typical user has 
his expected search length reduced from the 5,000 documents to be expected 
from a random search down to only ten documents, his search efforts will 
usually meet with early success. On the other hand, in a large collection 
of, say, 1,000,000 documents, it is questionable whether a reduction 
factor of only 99.8 per cent would be worth striving for. Even if optimal 
use were made of all nonstandard clues, there v;ould 'still be an expected 
search length in the neighborhood of 1,000 documents--a formidable number 
except for the more patient or desperate of the. library patrons. This is 
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not to say that all requesters would have so large an expected search 
length. Those of them who were able to supply an above-average amount 
of nonstandard information with above-average confidence in its accur- 
acy could expect better retrieval results. However, a minority would 
find the special facilities to be of real value. One is led to the 
tentative conclusion that the nonstandard information which could be 
supplied by a patron who has had previous contact with a document would 
by Itself usually be adequate in small collections, but inadequate in 
large collections, for the location of the document with a modicum of 
search effort on the patron's part. 

If the patron can remember some fragment of standard information 
in addition to the nonstandard information, the picture changes. Suppose, 
for example, that he is able to remember that the work he wants comes 
under the subject heading "Psychology," but he can remember neither author 
nor title, nor can he supply more specific subject headings. A library 
of a million documents might have, say, 10,000 documents under this 
heading, in which case his single piece of standard information would 
by itself be hopelessly inadequate for retrieval purposes. Retrieval 
could nevertheless be effected in most such instances if the patron were 
able to supplement this subject heading information with nonstandard 
information; for his subject heading information in effect makes it a 
small collection which must be searched instead of a large one. The 
same would hold true of any other fragment of standard information which 
could be used to "narrow down" a large collection prior to a search 
using nonstandard information. Nonstandard information which is by 
itself inadequate for retrieval may become adequate when supplemented by 
whatever fragments of standard information are available, and this obser- 
vation tempers somewhat the pessimism of the previous conclusion stating 
that nonstandard information would usually be useless for retrieval in 
very large collections. The reader is referred to [3] for data on the 
types and amounts of fragmentary author-title-subjcct information which 
arc likely to bo remembered by patrons who have had previous contact 

with a document. 
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11. Suinmarv 

The two most significant findings of the research were the 
following: 

1 . 




2 . 



amount bv which the ave rage specific work requester's expectM 
search effort could he reduced by providing him with an appropriate . 
retrieval avatem to ex p loit his nonstandard information is a rati ^ 
en the order of 500 to one . (This very approximate figure assumes 
that the nonstandard information to be elicited from the user is 
somewhat more extensive than that investigated in the experiment. 

It is understood that the figure has relevance only to the re- 
quester of a specific work who is already acquainted with the 
physical document but cannot remember enough standard author- 
title-subject information about it to look it up on this basis 
a lorn . ) Snob a ratio would normall y be a dequate for convenie nt 
r.,rrinv.-i1 in small document c ollections, but inadequate in larg e 
..neeMons cvcont in cases w h e re t he nonstan ja rd information c a n _be 
s„ppl,.mented bv s ome fragment of standard informatio n- 

IL interesting to speculate whether the retrieval usefulness 
of nonstandard information could be enhanced by allowing each reported 
piece of nonstandard information to be aceompanied by an indication of the 
degree of confidence which the person reporting it has of its accuracy. 
Tliis could be done, for example, by means of a three-choice supplementary 
question asking him whether ho is "certain, fairly sure, or 
.all sure" of the accuracy of the answer he has given to a main question 
about a nonstandard feature. This supplementary information would in- 
troduce another level of complexity into all probability calculations, 
but might increase retrieval effectiveness considerably. There are also 
other additions and refinements which would probably improve the list of 
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twenty-four questions considered in this paper; for example, one could 
add questions on loan status (e.g., "Is the book a two-day book or a 
two-week book?"), on past circulation data (e.g., "Have you taken the 
book out before?" "If not, can you name anyone who has?"), and on a 
number of other physical characteristics such as weight or thickness. 

A few additional questions beside the twenty-four were in fact asked in 
the experiment; the results are reported in [3]. A final possibility 
for improving the retrieval effectiveness of nonstandard information 
would be to bring circulation data into play to further improve the 
prior probability estimates used in the document ranking presented as 
final output to the user. A simple rule for so doing would be: Within 

each rank of the ranking established on the basis of the nonstandard 
information, order the individual documents according to the number of 
times each has been withdrawn from the library within the past year. 

More sophisticated rules of greater potential value could obviously be 
formulated, and there is hope that a very substantial further reduction 
in expected search length could be achieved through such methods. It 
is especially recommended that this last possibility be investigated in 
any future research on the subject. 
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PART I 

1. Please fill in the information requested on the next page. Then 
look over the list of books and choose ten you think you would like 
to look at. Do not choose any book that you have already read or 
heard of before, and do not spend more then three or four minutes 
deciding which books to look at. 

2. Once you have chosen ten books, find them in the right-hand book 
stack (which is arranged alphabetically by authors) and take them 
to the desk. Be sure that you have gotten the right books before 
you go on. 

3. Examine each book as carefully as you can in the 45 minutes allot- 
ted for the rest of the session (both no. 3 and no. 4). Grade each 
book according to how interesting it is to you, and put it in one 
of the following categories; 

A. Very interesting, and you would like to read it. 

B. Interesting 

C. Dull 

D. Very dull, and you would not like having to read it. 

Mark on the list which category you assign to each book. Do this 
for the entire ten. 

4. Then pick out five books and write a comment or phrase character- 
ising a part of each book that you found particularly interesting 
or outstanding. Your comment should be a full identification of 
your interest. That is, it should be understandable to someone who 
does not have the book before them. Write your comments on the 
list in the space left under each book. 

When you have finished, return these instructions and the list. 
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PART I 



Name 

Academic Status 

Subject Field 

Previous contact with psychology (courses, etc.) 



Adler, Mortimer J. 



What Man Has Made of Man 



Baudouin, Charles 



Psychoanalysis and Aesthetics 



Bier, William C., ed . 



The Adolescent; his search for under- 
s tand ing 



Bull, Nina 



The Attitude Theory of Emotion 



Crane, Edward Villeroy My Minds and I 



David, Henry P., ed. Perspectives in Personality Theory 

Bracken, Helmut von, ed. 



Eaton, Joseph W. 



Culture and Mental Disorders 



Fiebleman, James K. 



Mankind Behaving; human needs and 
material culture 



Gates, Georginia Stickland The Modern Cat 



o 



Griffiths, Ruth 



The Abilities of Babies 
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Helleberg, Carl Gustaf, ed. 


Spirit Communications; a book written by the 
spirits of the so-called dead... 


Isaacson, Robert L. 
Hutt, Max L. 
Blum, Milton L. 


Psychology; the science of behavior 


Kellogg, W. N. and L. A. 


The Ape and the Child 


Laslett, Peter, ed. 


The Physical Basis of Mind 


Lewis, Don 


Quantitative Methods in Psychology 


Marcuse, F. L. 


Hypnosis; fact and fiction 


Messick, Samuel 
Ross, John 


Measurement in Personality and Cognition 


Munn, Norman L. 


Psychology; the fundamentals of human 
adjustment 


Oliver, John Rathbone 


Fear; the autobiography of James Edwards 


Bidding ton, Ralph 


The Psychology of Laughter 


Pratt, Carroll C. 


1 

The Logic of Modern Psychology 



- 128 - 



Richet, Charles 


Idiot Man 


Rogers, Carl R. 


Counseling and Psychotherapy 


Sidis, Boris 


The Psychology of Suggestion 


Stern, George G. 
Stein, Morris I. 
Bloom, Benjamin S. 


Methods in Personality Assessment 


Thomas, John F. 


Beyond Normal Cognition 


Tolman, Edward Chace 


Drives Toward War 


Von Fange, Eugene K. 


Professional Creativity 


Weinberg, Henry 
Hire, William A. 


Case Book in Abnormal Psychology 


Wolberg, Lewis R. 


The Technique of Psychotherapy 
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PART II of this expsriement tests recall of information about 
the books you examined during Part I of the experiment. You will be 
given a series of tests to see how much you can remember about these 
books. We are interested in finding out what kinds of things are re- 
membered about books, what types of clues aid memory, and what charac- 
teristics of books are most memorable. 

Three separate sections, each presenting a slightly modified 
approach, make up this part. As you finish each section, notify the 
examiner that you have done so, and you will be given the next section. 
Take as much time as you need; however, once you have indicated that 
you have completed a section, please do not make any changes, even in- 
significant ones (e.g. spelling, additions, substitutions, punctua- 
tion) in your written responses to the previous sections. 

Be sure you understand the directions before beginning each 

section. 

Note: After Part II, there will be another part to the exper- 

iment . 
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PART II 

1. On the cards provided, please reconstruct the comments or descrip- 
tive phrases you made for the five of the ten books you examined 
during Part I of this experiment. With each comment, please re- 
cord everything you can remember, even though vague, fragmentary, 
or seemingly unimportant, about that book. 

If you are unable to reconstruct your comment, but can remember 
some information about the book, please record this information 
also. f 

Please number the cards from one to five, and use the pen provided. 



* 






4 
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PART II 



The following checklists represent some possible characteristics of 
books. Using one checklist for each of the five books, please in- 
dicate all characteristics which you can now remember about each 
book but which you ware unable to remember, or which did not 
to you in the previous section. Your answers from Section 1 should 
have been returned to you; if they were not, please ask for them. 

If you wish to change any of your answers from Section 1, please 
record the change on the checklist, and not on the answer cards. 
Please number the checklists to correspond with the answer cards, 
and keep together the cards and checklists that describe the same 
book. Also, please use the pen provided. 

Fill in or check the appropriate information on the checklist, try- 
ing to describe the book as well as you can. The following are a 
series of special instructions on filling out the checklist. 

1. Author no special instructions 

2. Title no special instructions 

3. Type of work. If you really feel that the book cannot be placed 
in „ single category, you may check as many as you think neces- 
sary to describe the book. 

4. Level of readers. As in the preceding section, check more than 
one category if you think it is really necessary. 

5. Subject no special instructions 

6. Physical characteristics 

Color If the book had more than one color, please check all 
the appropriate colors on the checklist. 

Size Samples of size and thickness are displayed. Please 

give both the s ize and thickness of the samples that are 
closest to the book you are describing. 

Number of pages An interval or span of pages is an acceptable 
description here. 

7. Location no special instructions 

8. Publication 

Date If you cannot remember the exact date, you can give an 
approximate date or an interval that covers the time in 
which the book was published. 

9. Bibliographic features. On this question you are asked to in- 
dicate yes or no for several characterist?xs . Please add a 
third column, "don't know," and mark it ^or those characteris- 
tics about which you remember nothing. 



-132- 



1PART II 
Name 



1, Author 



Do you know any details about the author that might help to identify 
him? 



2. Title 



If you think this is not the exact title, what other possibilities 
may be correct?^ 

3. Type of work. Please indicate the category you think best describes 
the work? 

Handbook 

General textbook 

^Introduction to a subject 

Monograph covering a single aspect of a field 

^Collection of various writings by one or a number of authors 

Biography, autobiography, or work chiefly about a person 

History of a subject or discipline 

Self-improvement, self-help, self-understanding, etc. 

Research report 

^Thesis or dissertation 

R eport of a single case study 
^Other. Please specif y 

4. Level of readers for which intended 

P opularization 

T reatment understandable to the general educated layman 

Work for beginning college students ' 

Work for advanced students or professionals 

pother. Please specify^^ 

5. Subject. (Describe as specifically as you can in a few words) 



Do you remember any specific details from the book, for example, 
chapters, topics, examples, discussions, incidents, etc.? Please 
describe 

6. Physical characteristics 

Color (Check the choice that most accurately describes the book) 



blue or blue-green ^black, dark grey, green 

brown or tan dark blue, etc. p urple or 

orange red violet 

.grey white yellow 
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If vou don't remember the exact color, wa«J the book 



a light 
color 
a dark 
color 



9 



Did the cover of the book have any characteristics other than a 
solid color and/or printing? Please check any descriptions that 

apply. 



two-color binding 
marbled cover 



patterned cover 
illustration on 
cover 



design of 
some kind 
on cover 



Type of printing 

library binding or 
rebinding 

spiral binding 

Other 



cloth publisher's 
binding 

library-reinf orced 
paperback 



paper-cov- 
ered board 
pamphlet 
binding 
leather 
binding 



Condition of book 

new somewhat usied ^bad 

fairly new worn 

Kind of illustrations (if any) 

Sme 1 1 — — — 

Approximate size — — 

Kind of printing — — 

Approximate number of pages^ — 

Typographical layout and arrangement of the book 



7. Location 

She 1 f 

Approximate location on shelf 

Call number 

Publication 

8. Date of publication 

Pub lisher 

Type of publishe r 
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Place of publication ^ — 

Name of series (if any) 

Translated? If so, from what language? 

Is the book a reprint or a revised edition? 

9. Bibliographic features. Please indicate whether each of the fol 
lowing is included in the book. 

YES NO 



index 

bibliography or suggested readings 
footnotes 

chapter titles (list any you remember) 

tables 

graphs 

problems at ends of chapters 

case studies 

glossary 

dedication If so, to whom? 

prefatory material 
quotations from literary works 
figures, charts, diagrams 

Is the book a single-volume book? 
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PART II 

3. The following is a list of the five comments you made during Part 
I of the experiment. WiHh these as a reminder, supplement or 
change the information you provided about each of the five books 
on the checklists you used previously. 

If these comments bring to mind one of the five books which you did 
not remember before, please fill in another checklist for this book. 

Please note beside the comments on this page the numbers of the 
cards and checklists that describe the book that goes with each com- 
ment. Or, if you cannot match these comments with the cards and 
checklists, please note this fact. 



Please use the pen provided. 
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PART II 

3a. After you have completed the three sections of Part II, please re- 
turn to the checklists used in Section 2 (use the blank space at 
the bottom of the last page) and: 

1) Compose what you would consider to be a good descriptive title 
for each book 

2) List a subject heading or headings under which you would look 
in order to locate the book in the Harper catalog. 



O 
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PART III 



1 Please examine the book list you are given and mark with a check 
Ihe ten books you examined during Part I of the experiment. Place 
a double check by the five books you commented on. 



♦ 
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AUTHOR-TITU: LIST 



Adler, Gerhard 



Adler, Mortimer J. 



Adrian, E.D. 



Alexander, Franz 



Allport, Gordon W. 



Anastasi, Anne 



Andrews, T.G., ed. 



Angell, James Rowland 



Angyal, Andras 



Arnold, Magda B. 



Baldwin, Alfred L. 



Barry, Frederick / 



Bartley, Samuel Howard 



Bass, Bernard M. & 

Irv;in A. Be'.'g, eds . 



Baudouin, Charles 



Beck, Samuel J. 



Beers, Clifford Whittingham 



Bennett, Margs^ret E. 



Bently, John Edward 
Berg, Charles 



Berg, Jan Hendrick Van den 



Berne, Eric 
Bernhardt, Karl S. 



The Living Symbol; a case study in the 
process of Individuation 
Social Interest, a Challenge to Mankind 
Topics in Psychology for Students of Law 
What Man Has Made of Man 
The Basis of Sensation 
The Social Life of Chimpanzees 
Fundamentals of Psychoanalysis 
The Psychoanalysis of the Total Person- 
ality 

Mental Maladjustments in Normal People 
Personality; a psychological interpre- 
tation 

Experimental Psychical Research 

Psychological Tasting 

Methods in Personality Assessment 

Methods of Psychology 

Human Behavior and Science 

An Introduction to Psychology 

Foundations for a Science of Personality 

The Fundamentals of Human Adjustment 

Hidden Remnants 

Story Sequence Analysis 

Behavior and Development In Childhood 

Moral Adjustment in Children 

The Scientific Habit of Thought 

Words as Traitors 

Beginning Experimental Psychology 

Psychology: a Behavioral Science 

Textbook of Abnormal Psychology 

Objective Approaches to Personality As- 
sessment 

Psychoanalysis and Aesthetics 
The World of Psychoanalysis 
The Human Senses 
Rorschach's Tests 
A Mind that Found Itself 
Techniques of Persuasion 
Guidance and Counseling in Groups 
New Directions in Psychology 
Superior Children 

The Normal Child and Some of His Abnor- 
malities 
Deep Analysis 
Inner Lights 

The Changing Nature of Man 
The Dynamics of the Psyche 
Games People Play 
Psychology of Laughter 
^he Art of Relaxation 
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Bier, William C., ed. 



Bindra, Dalbir 



Bischof, L.J. 



Blake, Robert R. & 

Glenn V. Ramsey 



Bovet, Richard 



Brennan, Robert Edward 



Briggs, Katharine May 



Broad, C.D. 



Broad ley, Margaret E. 



Brown, Norman Oliver 



Buckham, John Wright 



Bull, Nina 



Butler, Judson Rea & 

Theodore F. Karwaske 



Cameron, Norman 



Cantril, Hadley & 

Charles Heath Bumstead 



Chase, Stuart 



Clarke, Edwin Leavitt 



Cobb, Stanley 



Cohen, Yehudi A. 
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PART III 

2. Look over the indicated book stack and see if you can find the ten 
books you examined during Part I of the experiment. Please do not 
look inside the books. 

Remove the books from the shelf, and then match up your cards and 
checklists with the books that they describe. 
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PART III 

The ten books have been moved back to their original places in the 
book stack from which you took them. Please see if you can find 
them. Take the books from the shelves, and then match up the cards 
and checklists with the books that they describe. 
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APPENDIX B 

BOOK-MEMORY EXPERIMENT DATA 
Rose A. Spaith 

The data presented in this appendix is the numerical data from 
which the tables in the text were derived. Data are arranged and numbered 
to correspond to checklist items regarding the various book characteristics 
Data are arranged in matrices with the columns representing actual con- 
ditions of book characteristics and the rows representing experimentees ’ 
responses. The underlined figures, which, with a few exceptions, occur at 
the intersections of column and row, represent correct responses. Null 
responses appear in a separate row. Responses which are neither underlined 
nor in the "Null" row are incorrect. 

The following characteristics are not presented in this appendix: 
Checklist items (1) and (2), Author and Title are fully presented in the., 
text; numerical data appear on pages 21 and 23. Items (6g)> (6k) and 
(6i), Smell, Kind of printing, and Typographical layout, were not tabulated 

because of the paucity of responses the items elicited. The few resporlses 
that were given were not descriptive of the nature of the characteristics.. 
Item (5), Subject, was the subject of a separate investigation, reported “in 
"Accuracy and Utility of Subject-Related Responses;" the numerical data 
are presented in that paper. 
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TABLE 3 (2) 

TYPE OF WORK 
SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


192 


43.6 


Incorrect 


243 


55.2 


Null 


5 


1.2 



Total 



440 



100.0 
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TABLE 4 (1) 

LEVEL OF READERS 
DISTRIBUTION OF RESPONSES 







Actual Level of 


Obiect Books 






Response 


Popular 


Generally 

Educated 

Layman 


Beginning 

College 

Student 


Advanced 

Student/ 

Professionals 


Total 


Popular 


U 


34 


2 


4 


52 


Generally educated 
layman 


5 


56 


11 


57 


129 


Beginning college 
student 


• • 


15 


4 


21 


40 


Advanced student/ 
professionals 


• • 


6 


10 


M 


104 


Multi-response 

correct 


1 




9 


26 


75 


Multi-response 

wrong 


• • 


4 


3 


23 


30 


Other 


• • 


2 


• • 


4 


6 


No response 


• • 


* • 


1 


3 


4 


Total 


22 


152 


40 


226 


440 



TABLE 4 (2) 

LEVEL OF READERS 
SUMMARY OF RESPONSES 



Oualitv of Response 


Number 


Percentage 


Correct 


235 


53.4 


Incorrect 


201 


45.7 


Null 


4 


.9 


Total 


440 


100.0 







TABLE 6-a (1) 



COLOR 

DISTRIBUTION OF RESPONSES 




TABLE 6-a (2) 
COLOR 

SUMMARY OF RESPONSES 



Oualitv of Response 


Number 


Percentage 


Correct 


83 


18.9 


Incorrect 


195 


44.3 


Null 


162 


36.8 


Total 


440 


100.0 



o 

ERIC 
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TABLE 6-b 

LIGHT OR DARK COLOR OF COVER 
DISTRIBUTION OF RESPONSES 



Response 


Actual Cover Shade 
of Obiect Books 


Total 


Light 


Dark 


Both^ 


Light 


33 


39 


9 


81 


Dark 


35 


114 


14 


163 


No response 


50 


122 


24 


196 


Total 


118 


275 


47 


440 



^A number of books had bi-colored bindings 
since both "light" or "dark" responses were prob- 
lematic, "light" was considered correct. 



LIGHT OR DARK COLOR OF COVER 
SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


156 


35.4 


Incorrect 


88 


20.0 


Null 


196 


44.6 


Tota 1 


440 


100.0 
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TABLE 6-c (1) 



COVER DETAIL 

DISTRIBUTION OF RESPONSES 



Response 


Actual 


Cover 


Detail 


of Object 


Books 


Total 


u 

o 

1—1 

0 
a 

1 

o 

5 

H 


TJ 

0) 

1—1 

CO 

s 


c 

u 

cu 

4J 

4J 

CO 

PM 


TJ 

CU 

4J 

CO 

U 

4J 

CO 

D 

I— 1 
I— 1 

M 


c 

WD 

•H 

CO 

CU 

Q 


bO 

C 

•H 

4 : 

4J 

0 

Z 


Two-color 


8 


• • 




• • 


2 


14 


24 


Marbled 


• • 


• • 




• • 


2 


2 


4 


Pattern 


• • 


• • 




• • 


. 2 


1 


3 


Illustrated 


1 


• • 




5 


2 


3 


11 


Design 


1 


• a 




1 


6 


7 


15 


Nothing^ 


6 


• • 




2 


11 


47 


66 


Other 


1 


• • 




• • 


1 


• • 


2 


No response 


42 


5 




2 


34 


232 


315 


Total 


59 


5 


• • 


10 


60 


306 


440 


^This 


response was 


written 


in 


by 66 


respondents . 



TABLE 6-c (2) 

COVER DETAIL 
SUMMARY OF RESPONSES 



Quality of Responses 


Number 


Percentage 


Correct 


66 


15.0 


Incorrect 


59 


13.4 


Null 


315 


71.6 


Total 


440 


100.0 
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TABLE 6-d (1) 

TYPE OF BINDING 
DISTRIBUTION OF RESPONSES 





Actual 


Binding 


of 1 


Obiec t 


Books 






Response 


(xO 

C CxO 
•H c 

C ^ 
•H C 
pq -H 

Q) 

^ Pi 
CO 

u u 

O 

•H 

h4 


CO 

u 

Q) 

CO 

•H 

f— 1 

txO 
3 c 

-H 

43 a 

JJ -H 
O PQ 

r— 1 

o 


bO 

C 

•H 

d 

•H 

PQ 

r— i 

CO 

•H 

CL. 

CO 


o; 

u 

u 

0 

c 

•H ^ 

0) u 

U CO 

1 

^ u 
u a; 

CO CX4 
U CO 
rO PL| 
•H 


o; 

u 

Q) 

> 

0 CO 
U T3 

1 U 
U CO 

cu o 
Cu pq 

CO 

Pi4 


bO 

C 

•H 

d 

•H 

PQ 

4J 

(U 

r-4 

• 43 

CL, 

e 

CO 

Pl4 


bO 

d 

•H 

d 

•H 

PQ 

}-l 

CU 

43 

4-» 

CO 

CU 


Total 


Library binding or 
rebinding 




40 


• • 


3 


2 


2 


• • 


67 


Cloth publisher's 
binding 


34 


246 


• • 


2 


13 


• • 


• • 


295 


Spiral binding 


• # 


1 


JL 


1 


• • 


• • 


• • 


3 


Library-reinforced 

paperback 


• • 


1 


•* • 


9 


• • 


• • 


• • 


10 


Paper-covered 

boards 


• • 


3 


• • 


0 


JL 


• • 


• • 


4 


Pamphlet binding 


• • 


• • 


• • 


• • 


1 


2 


• • 


3 


Leather binding 


• • 


1 


• » * 


• • 


1 


• • 


• • 


2 


Other 


• • 


4 


• % 


• • 


2 


• • 




6 


No response 


7 


35 


• • 


1 


6 


• • 


1 


50 


Total 


61 


331 


1 


16 


26 


4 


1 


440 



TABLE 6-d (2) 

TYPE OF BINDING 
SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


279 


63.5 


Incorrect 


111 


25.2 


Null 


50 


11.3 



Total 



440 



100.0 



TABLE 6-e 



CONDITION 

DISTRIBUTION OF RESPONSES 



Response 


Actual Condition of 
Object Books 


Total 


New 


Fairly 

New 


Used 


Worn 


Bad 


New 


23 


18 


11 


4 


3 


59 


Fairly new 


19 


38 


52 


13 


15 


137 


Used 


7 


12 


6 ^ 


49 


30 


162 


Worn 


1 


3 


26 


17 


17 


64 


Bad 


• • 


• • 


• • 


• • 


1 


1 


No response 


1 


5 


6 


5 


• • 


17 


Total 


51 


76 


159 


88 


66 


440 



CONDITION 





SUMMARY OF 


RESPONSES 




Quality of 


Response 


Number 


Percentage 


Correct 




143 


32.2 


Incorrect 




280 


63.6 


Null 




17 


4.1 


Total 




440 


100.0 
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TABLE 6-f 
ILLUSTRATIONS 
DISTRIBUTION OF RESPONSES 




SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


251 


57.1 


Incorrect 


56 


12.7 


Null 


133 


30.2 



Total 



440 



100.0 
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TABLE 6-h (1) 
HEIGHT^ 

DISTRIBUTION OF RESPONSES 



Response 






Actual 


Height of Obiect 


Books 






Total 


7 


7>s 


8 


8^ 


9 


9^ 


,10 


lOh 


11 


Ilk 


5 


1 


• • 


• • 


3 


• • 












4 


5% 






















0 


6 


2 


• • 


5 


2 


3 












12 


6^ 






















0 


7 


9 


6 


21 


18 


13 












67 


1\ 


• • 


J, 


2 


4 


1 












8 


8 


2 


1 


28 


54 


35 


3 










123 


8^ 


1 


• • 


1 


9 


3 


• • 










14 


9 


1 


2 


8 


26 


30 


8 


3 








78 


9^ 






• • 


2 


1 


2 


1 








6 


10 






1 


6 


10 


4 


6 








27 


10^ 






















0 


11 






2 


3 


4 


2 


5 




4 




20 


11^2 






1 
















1 


12 






1 


• • 


1 


1 


3 








6 


14 












1 


• • 








1 


No response 


4 


4 


17 


21 


20 


4 


3 








73 


Total 


20 


14 


87 


148 


121 


25 


21 




4 




440 


^Heights and 


responses 


are 


shown 


to 


the 


nearest 


one 


-half 


inch. 



TABLE 6-h (2) 
HEIGHT 

SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


96 


21.8 


Incorrect 


307 


69.8 


Null 


37 


8.4 


Total 


440 


100.0 
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TABLE 6-j (1) 

NUMBER OF PAGES 
DISTRIBUTION OF RESPONSES 









Actual 


Number o 


f Pages 


of 


Object 


Books 






Total 


Response 


<jr 

1 

O 


1 

o 

m 


T— < 
1 

o 

o 

T— < 


CTi 

T— < 
1 

o 

m 

T— < 


CM 

1 

O 

o 

CM 


CTi 

CTi 

CM 

1 

O 

m 

CM 


CO 

1 

o 

o 

CO 


CTi 

CO 

1 

O 

m 

CO 


<t 

1 

O 

o 

<t 


CTi 

(Ti 

<t 

1 

O 

m 


m 

1 

o 

o 

m 


+ 

o 

m 

m 


0-49 


2 


• • 




















• • 


2 


50-99 


5 


4 


3 


• • 


















12 


100-149 


1 


3 


13 


3 


5 


3 


• • 










• • 


28 


150-199 


• • 


4 


10 


9 


11 


7 


5 


3 


3 


1 


• 

• • 


• • 


53 


200-249 






7 


13 


2^ 


15 


8 


4 


5 


1 


• • 


1 


74 


250-299 






2 


4 


16 


16 


5 


2 


8 


2 


1 


2 


58 


300-349 






4 


6 


12 


9 


12 


7 


7 


4 


1 


3 


65 


350-399 






• • 


4 


5 


6 


7 


5 


8 


4 


• • 


• • 


39 


400-449 






• • 


3 


4 


1 


1 


3 


1 


4 


• • 


3 


24 


450-499 










1 


2 


• • 


4 


3 


J, 


• • 


3 


14 


500-549 










1 


• • 


3 


• • 


5 


3 


• • 


4 


16 


550+ 












• • 


1 


• • 


6 


2 


• • 


9 


18 


No response 


2 


• • 


5 


3 


4 


6 


4 


4 


6 


1 


• • 


2 


37 


Total 


10 


11 


44 


45 


79 


65 


46 


32 


56 


23 


2 


27 


440 





TABLE 6-j (2) 






NUMBER OF 'PAGES 






SUMMARY OF RESPONSES 




Quality 0 


£ Responses Number 


Percentage 


Correct 


96 


21.8 


Incorrect 


307 


69.8 


Null 


37 


8.4 



Total 



440 



100.0 



-167- 



table 7- a 

POSITION IN BOOKSTACK 
DISTRIBUTION OF RESPONSES 



Response 


Actual Position of 
Object Book 


Total 


High 


Middle 


Low 


High 


31 


7 


8 


46 


Middle 


16 


15 


31 


82 


Low 


3 


8 


26 


37 


No response 


78 


102 


95 


275 


Total 


128 


152 


160 


440 



POSITION IN BOOKSTACK 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


92 


20.9 


Incorrect 


73 


16.6 


Null 


275 


62.5 



Total 



440 



100.0 
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TABLE 7-b 
POSITION ON SHELF 
DISTRIBUTION OF RESPONSES 



Response 


Actual Position of Obiect Book 


Total 


Left 


Middle 


Right 


Left 




3 


9 


26 


Middle 


14 


11 


5 


30 


Right 


10 


6 


11 


33 


No response 


133 


115 


103 


351 


Total 


171 


135 


134 


440 



SUMl^RY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


42 


9.5 


Incorrect 


47 


10.7 


Null 


351 


79.8 


Total 


440 


100.0 



TABLE 7-c 
CALL NUMBER 
SUMMARY OF RESPONSES 


Quality of 


Response 


Number 


Percentage 


Correct 




17 


3 .'9 


Incorrect 




6 


1.3 


Null 




417 


94.8 


Total 


440 


100.0 



O 
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TABLE 8-a (1) 

DATE OF PUBLICATION 
DISTRIBUTION OF RESPONSES 



Response 


Actual 


Date 


of Publication 


of Object 


Books 




Total 


CO 

o 

VO 

C3^ 

1-1 


CO 

o 

m 

f— < 


CO 

o 

<T\ 


CO 

o 

CO 


CO 

o 

<N 

f— < 


CO 

o 

f— < 
I— < 


CO 

o 

o 

f— < 


CO 

o 

00 
f— < 


o 

CO 

r— 1 
1 

0) 

u 

PM 


u 

0) 

u 

o 


1960 's 




16 


3 


• • 












• • 


69 


1950 's 


37 




19 


12 


1 


• • 








• • 


109 


1940 's 


6 


15 


le 


10 


• • 


• • 


2 


• • 




• • 


49 


1930 's 


1 


10 


6 


il 


6 


• • 


1 


1 




1 


43 


1920' s 


1 


3 


2 


4 


4 


1 


• • 


• • 




• • 


15 


1910's 


1 


1 


3 


1 


1 


3 


• • 


1 




• • 


11 


1900 's 


• • 


1 








1 


1 


• • 




• • 


3 


1890's 


• • 


• • 


• • 


1 


• • 


• • 


• • 


1 




• • 


2 


Pre-1890 


1 


• • 


• • 


3 


1 


• • 


^ • • 


• • 


2 


2 


9 


Other 


5 


2 


1 


1 


• • 


1 


1 


• • 


1 


• • 


12 


No response 


34 


40 


26 


10 


4 


2 


1 


• • 


• • 


1 


118 


Total 


136 


128 


76 


59 


17 


8 


6 


3 


3 


4 


440 



TABLE 8-a (2) 

DATE OF PUBLICATION 
SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


134 


30.4 


Incorrect 


188 


42.8 


Null 


118 


26.8 


Total 


440 


100.0 
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TABLE 8-b 
PUBLISHER 

SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


15 


3.5 


Incorrect 


17 


3.8 


Null 


408 


92.7 


Total 


440 


100.0 



TABLE 8-c 

TYPE OF PUBLISHER 
SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


40 


9.1 


Incorrect 


40 


9.1 


Null 


360 


81.8 


Total 


440 


100.0 



TABLE 8-d 

PLACE OF PUBLICATION 
SUMMARY OF RESPONSES 


Quality of Responses 


Number 


Percentage 


Correct 


29 


6.6 


Incorrect 


72 


16.4 


Null 


339 


77.0 


Total 


440 


100.0 



r 
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TABLE 8-e 
SERIES 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descrii 


ption of Book 


Total 


Series ' 


Not a Series 


Series 


7 


3 


10 


Not a series 


7 


44 


51 


Null 


74 


305 


379 


Total 


88 


352 


440 



summary of responses 



Quality of Response 


Number 


Percentage 


Correct 


51 


11.6 


Incorrect 


10 


2.3 


Null 


379 


86.1 


Total 


440 


. 100.0 



TABLE 8-f 

translation 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri] 


ption of Book 


Total 


Translation 


Not a Translation 


Translation 


21 


12 


33 


Not a translation 


25 


267 


292 


Don ' t know 


14 


101 


115 


Total 


60 


390 


440 



SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


288 


65.4 


Incorrect 


37 


8.4 


Null 


115 


26.2 


Total 


440 


100.0 



ERIC 



-172- 



TABLE 8-g 

REPRINT OR REVISION 
DISTRIBUTION OF RESPONSES 





Actual Descri 


ption of Book 


• 


Response 


Reprinted or 
Revised Edition 


Not Reprinted Or 
Revised Edition 


Total 


Reprinted or revised 
edition 


16 


8 


24 


Not reprinted or rev. 


32 


183 


215 


Don't know 


36 


165 


201 


Total 


84 


356 


440 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


199 


45.2 


Incorrect 


40 


9.1 


Null 


201 


45.7 


Total 


440 


100.0 



TABLE 9-a ' 

INDEX 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri] 


ption of Book 


Total 


Index 


No Index 


Index 


155 


57 


212 


No index 


50 


37 


87 


Don ' t know 


96 


45 


141 


Total 


301 


139 


440 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


192 


43.6 


Incorrect 


107 


24.4 


Null 


141 


32.0 


Total 


440 


100.0 
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TABLE 9-b 
BIBLIOGRAPHY 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descrij 


ption of Book 


Total 


Bibliography 


No Bibliography 


Bibliography 
No bibliography 
Don't know 

Total 


76 66 
42 66 
100 90 

218 222 


142 

108 

190 

440 



SUMMARY OF 


RESPONSES 




Oualitv of Response 


Number 


Percentage 


Correct 


142 


32.3 


Incorrect 


108 


24.5 


Null 


190 


43.2 


Total 


440 


100.0 



♦ 



4 



TABLE 9-c 



FOOTNOTES 



DISTRIBUTION OF RESPONSES 



Response 


Actual Description of Book I 


Total 


Footnotes 


No Footnotes 


Footnotes 
No footnotes 
Don' t know 

Total 


122 29 

79 59 

105 46 

306 134 


151 

138 

151 

440 



SUMMARY OF RESPONSES 



Quality of Response 

Correct 

Incorrect 

Null 

Total 



Number Percentage 



181 


41.1 


108 


24.6 


151 


34.3 


440 


100.0 



1 







-174- 
table 9-d 

CHAPTER TITLES 
DISTRIBUTION OF RESPONSES 





Actual Descri 


ption of Book 




Response 


Chapter Titles 


No Chapter Titles 


Total 


Chapter titles 


350 


10 


360 


No chapter titles 


22 


7 


29 


Don't know 


49 


2 


51 


Total 


421 


19 


440 



SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


357 


81.1 


Incorrect 


32 


7.3 


Null 


51 


11.6 


Total 


440 


100.0 






TABLE 9-e 
TABLES 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri] 


ption of Book 


Total 


Tables 


No Tables 


Tables 


48 


24 


72 


No tables 


43 


213 


256 


Don ' t know 


36 


76 


112 


Tota 1 


127 


313 


440 



SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


261 


59.3 


Incorrect 


67 


15.2 


Null 


112 


25.5 


Total 


440 


100.0 




JL :,L 
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TABLE 9-f 
GRAPHS 



DISTRIBUTION OF RESPONSES 



Response 


Actual Descri 


ption of Book 


Total 


Graphs 


No Graphs 


Graphs 
No graphs 
Don't know 

Total 


24 25 

25 243 

39 84 

88 352 


49 

268 

123 

440 



SUMMARY OF RESPONSES 



Oualitv of Response 


Number 


Percentage 


Correct 


267 


60.7 


Incorrect 


50 


11.3 


Null 


123 


28.0 


Total 


440 


100.0 



TABLE 9-g 
PROBLEMS 



DISTRIBUTION OF RESPONSES 




Correct 

Incorrect 

Null 



358 


81.4 


15 


3.4 


67 


15.2 



Total 



440 



100.0 
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table 9-h 

CASE STUDIES 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri 


ption of Book 


Total 


Case Studies 


No Case Studies 


Case studies 


119 


70 


189 


No case studies 


17 


120 


137 


Don ' t know 


30 


84 


114 


Total 


166 


274 


440 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


239 


54.3 


Incorrect 


87 


19.8 


Null 


114 


25.9 


Total 


440 


100.0 



TABLE 9-i 
GLOSSARY 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descrii 


ption of Book 


Total 


Glossary 


No Glossary 


Glossary 


4 


21 


25 


No glossary 


7 


240 


247 


Don' t know 


10 


158 


168 


Total 


21 


419 


440 



SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


244 


55.4 


Incorrect 


28 


6.4 


Null 


168 


38.2 


Total 


440 


100.0 



-177- 



TABLE 9-j 
DEDICATION 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri] 


ption of Book 


Total 


Dedication 


No Dedication 


Dedication 


38 


33 


71 


No dedication 


17 


41 


58 


Don ' t know 


141 


170 


311 


Total 


196 


244 


440 



SUMMARY OF 


RESPONSES 




Ouality of Response 


Number 


Percentage 


Correct 


79 


17.9 


Incorrect 


50 


11.4 


Null 


311 


70.7 


Total 


440 


100. 0 



TABLE 9-k 
PREFACE 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descrii 


Ption of Book 


Total 


Preface 


No Preface 


Preface 


240 


16 


256 


No preface 


20 


7 


27 


Don't know 


143 


14 


157 


Total 


403 


37 


440 



SUMMARY OF RESPONSES 



Quality of Response 


Number 


Percentage 


Correct 


247 


56.1 


Incorrect 


36 


8.2 


Null 


157 


35.7 


Total 


440 


100.0 
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table 9-1 

QUOTATIONS 

DISTRIBUTION OF RESPONSES 



Response 


Actual Descri] 


ption of Book 


Total 


Quotations 


No Quotations 


Quotations 
No quotations 
Don' t know 

Total 


52 32 

35 126 

55 1^*1 

142 298 


84 

161 

195 

440 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


178 


40.5 


Incorrect 


67 


15.2 


Null 


195 


44.3 


Total 


440 


100.0 



TABLE 9-m 

FIGURES, CHARTS, DIAGRAMS 
DISTRIBUTION OF RESPONSES 




SUMMARY OF RESPONSES 



Quality of Response 

Correct 

Incorrect 

Null 

Total 



Number Percentage 



267 


60.7 


55 


12.5 


118 


26.8 


440 


100.0 
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TABLE 9-n 
SINGLE VOLUME 



DISTRIBUTION OF RESPONSES 



Response 


Actual Description of Book 


Total 


Single Volume 


Part of 

Multi- volume Set 


Single volume 
Part of multi- 
volume set 
Don' t know 

Total 


411 2 

9 8 

8 2 

428 12 


413 

17 

10 

440 



SUMMARY 


OF RESPONSES 




Quality of Response 


Number 


Percentage 


Correct 


419 


95.2 


Incorrect 


11 


2.5 


Null 


10 


2.3 


Total 


440 


100.0 



o 



APPENDIX C 



COMPARATIVE DATA FOR PSYCHOLOGY AND 
NON-PSYCHOLOGY EXPERIMENTERS 



TABLE 1 
AUTHOR 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


59 


16.2 


Incorrect 


19 


5.2 


Null 


286 


78,6 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


79 


21.7 


Incorrect 


219 


60.1 


Null 


66 


18.2 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


173 


47.4 


Incorrect 


186 


51.2 


Null 


5 


1.4 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


11 


14.5 


Incorrect 


3 


3.9 


Null 


62 


81.6 


Total 


76 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


21 


27.6 


Incorrect 


34 


59.2 


Null 


10 


13.2 


Total 


76 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


19 


25.0 


Incorrect 


57 


75.0 


Null 


0 


0.0 


Total 


76 


100.0 



TABLE 2 
TITLE 

SUMMARY OF RESPONSES 



Non-Psychology 



TABLE 3 
TYPE OF WORK 
SUMMARY OF RESPONSES 

Non-Psychology 



TABLE 4 

LEVEL OF READER 







SUMMARY OF 


RESPONSES 








Non-Psychology 






Psychology 




Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 

Incorrect 

Null 


190 

170 

4 


52.3 

46.8 

.9 


Correct 

Incorrect 

Null 


45 

31 

0 


59.2 

40.8 

0.0 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 5 

SUBJECT DESCRIPTION 
SUMMARY OF RESPONSES 



Non-Psychology 




Psychology 




Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


270 


74.2 


Correct 


63 


82.9 


Incorrect 


92 


25.2 


Incorrect 


13 


17.1 


Null 


2 


.6 


Null 


0 


0.0 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 5a 

SUBJECT HEADINGS^ 
SUMMARY OF RESPONSES 



Non-Psychology 



* ■ ' ' ' * 

Quality of 
Response 


Number 


Percentage 


Correct 


98 


70.0 


Incorrect 


37 


26.4 


Null 


5 


3.6 


Total 


140 


100.0 



Psychology 


Quality of 
Response 


Number 


Percentage 


Correct 


38 


73.1 


Incorrect 


14 


26.9 


Null 


0 


0.0 


Total 


52 


100.0 



^Responses from Subject Heading question (SH question). 



r 
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TABLE 6a 
COLOR 

SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality of 

G non ^ P 


Number 


Percentage 


Quality of 
Resnonse 


Number 


Percentage 


Correct 

Incorrect 

Null 

Total 


64 

167 

133 

364 


17.5 
• 46.0 

36.5 

ioo.o 


Correct 

Incorrect 

Null 

Total 


19 

28 

29 

76 


25.0 

36.8 

38.2 

100.0 



Non-Psychology 


TABLE 6b 
LIGHT/DARK 
SUMMARY OF RESPONSES 


Psychology 




Quality of 

PoGnonRP 


Number Percentage 


Quality of 
Response 


Number 


Percentage 


|\c; D pwLloc 

Correct 

Incorrect 

Null 

Total 


131 

77 

156 

364 


36.0 

21.0 
43.0 

100.0 


Correct 

Incorrect 

Null 

Total 


25 

11 

40 

76 


• 32.9 

14.4 
52.7 

100.0 




Non-Psychology 


TABLE 6c 
COVER DETAIL 
SUMMARY OF RESPONSES 


Psychology 




Quality of 

PoGnnn P 


Number 


Percentage 


Quality of 
Resnonse 


Number 


Percentage 


jLlitw O » III o ^ 

Correct 

Incorrect 

Null 

Total 


54 

51 

259 

364 


14.8 

14.0 

71.2 

100.0 


Correct 

Incorrect 

Null 

Total 


12 

8 

56 

76 


15.8 

10.5 

73.7 

• 100.0 
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Non-Psychology 


TABLE 
TYPE OF 
SUMMARY OF 


6d 

BINDING 

RESPONSES 


Psychology 




Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


234 


64.3 


Correc t 


45 


59.2 


Incorrect 


97 


26. / 


Incorrect 


14 


18.4 ■ 


Null 


33 


9.0 


Null 


17 


22.4 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 6e (ie E) 
CONDITION 

SUMMARY OF RESPONSES 



Non-Psychology 




Psychology 




Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


108 


29.6 


Correct 


34 


44.7 


Incorrect 


240 


66.0 


Incorrect 


40 


52.7 


Null 


16 


4.4 


Null 


2 


2.6 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 6f 
ILLUSTRATIONS 
SUMMARY OF RESPONSES 

Non-Psychology 


Psychology 




Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


204 


56.0 


Correc t 


47 


61.8 


Incorrect 


45 


12.4 


Incorrect 


11 


14.5 


Null 


115 


3' .6 


Null 


18 


23.7 


Total 


364 


100.0 


Total 


76 


100.0 
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TABLE 6h 
SIZE 

SUMMARY OF RESPONSES 



Non-Psychology 



Psychology 



Quality of 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


KGSponbc 

Correct 

Incorrect 

Null 


76 

227 

61 


20.8 

62.4 

16.8 


Correc t 

Incorrect 

Null 


13 

51 

12 


17.2 

67.0 

15.8 


Total 


364 


100.0 


Total 


76 


100.0 






TABLE 


6j 










NUMBER OF 


PAGES 










SUMMARY OF 


RESPONSES 






Non-Psychology 




Psychology 




Quality of 
ResoonSG 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 

Incorrect 

Null 


77 

253 

34 


21.1 

69.5 

9.4 


Correc t 

Incorrect 

Null 


19 

54 

3 


25.0 

71.0 
4.0 


Total 


364 


100.0 


Total 


76 


100.0 






TABLE 7a 










SHELF NUMBER 










SUMMARY OF RESPONSES 








Non-Psychology 




Psychology 




Quality of 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


KG spono c 

Correct 
Incorrec t 
Null 


73 

60 

231 


20.0 

16.5 

63.5 


Correct 

Incorrect 

Null 


19 

13 

44 


25.0 

17.1 
57.9 


Total 


364 


100.0 


Total 


76 


100.0 
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TABLE 7b 
SHELF POSITION 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


30 


8.2 


Incorrec t 


41 


11.3 


Null 


293 


80.5 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


12 


15.8 


Incorrect 


• 6 


7.9 


Null 


58 


76.3 


Total 


76 


100.0 



TABLE 7c 
CALL NUMBER 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


10 


2.7 


Incorrect 


5 


1.4 


Null 


349 


95.9 


Tota 1 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


7 


9.2 


Incorrect 


1 


1.3 


Null 


68 


89.5 


Total 


76 


100.0 



TABLE 8a 

DATE OF PUBLICATION 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


112 


30.8 


Incorrec t 


155 


42.6 


Null 


97 


26.6 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


22 


29.0 


Incorrect 


33 


43.4 


Null 


21 


27.6 


Total 


76 


100.0 



O 



TABLE 8b 
PUBLISHER 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
ResDonse 


Number 


Percentage 


Correct 


15 


4.1 


Incorrect 


16 


4.4 


Null 


333 


91.5 


Total 


364 


100.0 





Psychology 




Quality of 
Response 


Number 


Percentage 


Correct 


0 


0.0 


Incorrect 


1 


1.3 


Null 


75 


98.7 


Total 


76 


100.0 



TABLE 8c 

TYPE OF PUBLISHER 



SUMMARY OF 

Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


38 


10.4 


Incorrect 


38 


10.4 


Null 


288 


79.2 


Total 


364 


100.0 



RESPONSES 



Psychology 


Quality of 
Response 


Number 


Percentage 


Correct 


2 


2.6 


Incorrect 


2 


2.6 


Null 


72 


94.8 


Total 


76 


100.0 



TABLE 8d 

place of publication 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


27 


7.4 


Incorrect 


68 


18.6 


Null 


269 


74.0 


Total 


364 


100.0 





Psychology 




Quality of 
Response 


Number 


Percentage 


Correct 


2 


2.6 


Incorrect 


4 


5.3 


Null 


70 


92.1 


Total 


76 


100.0 
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TABLE 8e 
SERIES 

SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality of 
ResDonse 


Number 


Percentage 


Correct 


48 


13.1 


Incorrect 


7 


1.9 


Null 


309 


85.0 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


3 


3.9 


Incorrect 


3 


3.9 


Null 


70 


92.2 


Total 


76 


100.0 






TABLE 8f 
translation 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


247 


68.0 


Incorrect 


33 


9.0 


Null 


84 


23.0 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


41 


53.9 


Incorrect 


4 


5.3 


Null 


31 


40.8 


Total 


76 


100.0 



TABLE Sg 
REPRINT/REVISION 



SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


168 


46.2 


Incorrect 


43 


9.2 


Null 


162 


44.6 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


31 


40.8 


Incorrect 


6 


7.9 


Null 


39 


51.3 


Total 


76 


100.0 
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TABLE 9a 
INDEX 



SUMMARY OF 

Non-Psychology 


RESPONSES 


Psychology 


Quality of 
ResoonsG 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


169 


46.5 


Correct 


23 


30.3 


Incorrect 


82 


22.5 


Incorrect 


25 


32.9 


Null 


113 


31.0 


Null 


28 


36 . 8 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 9b 
bibliography 



SUMMARY OF 

Non-Psychology 


RESPONSES 


Psychology 


Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


121 


33.2 


Correct 


21 


27.6 


Incorrect 


90 


24.8 


Incorrect 


18 


23.7 


Null 


153 


42.0 


Null 


37 


48.7 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 9c 
FOOTNOTES 



SUMMARY OF 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


156 


42.8 


Incorrect 


92 


25.3 


Null 


116 


31.9 


Total 


364 


100.0 



RESPONSES 





Psychology 




Quality of 
Response 


Number 


Perdentage 


Correct 


25 


32.9 


Incorrect 


16 


21.1 


Null 


35 


46.0 


Total 


76 


ipo.o 
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TABLE 9d 
CHAPTER TITLES 
SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


292 


80.2 


Incorrect 


27 


7.4 


Null 


45 


12.4 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


65 


85.5 


Incorrect 


5 


6.6 


Null 


6 


7.9 


Total 


76 


100.0 



table 9e 

TABLES 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


210 


57.6 


Incorrect 


58 


15.6 


Null 


96 


26.4 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


51 


67.1 


Incorrect 


9 


11.8 


Null 


16 


21.1 


Total 


76 


100.0 



TABLE 9f 
GRAPHS 

SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality bf 
Response 


Number 


Percentage 


Correct 


216 


59.3 


Incorrect 


41 


11.3 


Null 


107 


29.4 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


51 


67.1 


Incorrect 


9 


11.8 


Null 


16 


21.1 


Total 


76 


L.00.0 



TABLE 9g 
PROBLEMS 

SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


287 


78.9 


Incorrect 


13 


3.5 


Null 


64 


17.6 


Total 


364 


100.0 



Quality of 
Response 


Number 


Percentage 


Correct 


71 


93.4 


Incorrect 


2 


2.7 


Null 


3 


3.9 


Total 


76 


100.0 



TABLE 9h 



CASE STUDIES 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


199 


54.7 


Incorrect 


73 


20.0 


Null 


92 


25.3 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


40 


52.7 


Incorrect 


14 


18.4 


Null 


22 


28.9 


Total 


76 


100.0 



TABLE 9i 
GLOSSARY 



SUMMARY 





Non-Psychology 


Quality of 
Response 


Number 


Percentage 


Correct 


204 


56.0 


Incorrect 


25 


6.9 


Null 


135 


37.1 



RESPONSES 

Psychology 

Quality of Number Percentage 
Response — ^ 

Correct 40 52.6 

Incorrect 3 4.0 

Null 33 43.4 

iOO.O 



Total 



364 



100.0 



Total 



76 
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table 9j 

DEDICATION 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


61 


18.7 


Incorrect 


43 


9.9 


Null 


260 


71.4 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


18 


23.7 


Incorrect 


7 


9.2 


Null 


51 


67.1 


Total 


76 


100.0 



TABLE 9k 

PREFATORY MATERIAL 
SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


207 


51.0 


Incorrect 


31 


8.5 


Null 


126 


34.5 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


40 


52.6 


Incorrect 


5 


6.6 


Null 


31 


40.8 


Total 


76 


100.0 



TALBE 91 
QUOTATIONS 
SUMMARY OF RESPONSES 



Non-Psychology Psychology 



Quality of 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


144 


39.6 


Correct 


34 


44.7 


Incorrect 


57 


15.6 


Incorrect 


10 


13.2 


Null 


163 


44.8 


Null 


32 


42.1 


Total 


364 


100.0 


Total 


76 


100.0 



TABLE 9m 
FIGURES 

SUMMARY OF RESPONSES 



Non-Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


217 


59.6 


Incorrect 


49 


13.5 


Null 


98 


26.9 


Total 


364 


100.0 



Psychology 



Quality of 
Response 


Number 


Percentage 


Correct 


50 


65.8 


Incorrect 


6 


7.9 


Null 


20 


26.3 


Total 


76 


100.0 



TABLE 9n 
SINGLE VOLUME 
SUMMARY OF RESPONSES 

Non-Psychology Psychology 



Quality bf 
Response 


Number 


Percentage 


Quality of 
Response 


Number 


Percentage 


Correct 


344 


94.5 


Correct 


75 


98.7 


Incorrect 


11 


3.0 


Incorrect 


0 


0.0 


Null 


9 


2.5 


Null 


1 


1.3 


Total 


364 


100.0 


Total 


76 


100.0 



APPENDIX D 



DATA SUPPLEMENT TO ”THE POTENTIAL USEFULNESS OF CATALOG 
ACCESS POINTS OTHER THAN AUTHOR, TITLE, 

AND SUBJECT" 



r 
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table la 



DATA FROM MEMORY EXPERIMENT CONCERNING COLOR OF COVER 
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TABLE 2a 

DATA FROM MEMORY EXPERIMENT CONCERNING TYPE OF BINDING 







Actual Type of Binding 






Total 


Remembered 
Type of 
Binding 


Library Binding 
or Rebinding 


Cloth Publisher's 
Binding 


Spiral Binding 


Library Reinforced 
Paperback 


Paper Covered 
Boards 


Pamphlet Binding 


1 

Leather Binding 


Other 1 


Library binding or 




















rebinding 


20 


40 


0 


3 


2 


2 


0 


0 


67 


Cloth publisher's 




















binding 


34 


246 


0. 


2 


13 


0 


0 


0 


295 


Spiral binding . . . 


0 


1 


1 


1 


0 


0 


0 


0 


3 


Library reinforced 




















paperback 


0 


1 


0 


9 


0 


0 


0 


0 


10 


Paper covered boards 


0 


3 


0 


0 


1 


0 


0 


0 


4 


Pamphlet binding . . 


0 


0 


0 


0 


1 


2 


0 


0 


3 


Leather binding . . . 


0 


1 


0 


0 


1 


0 


0 


0 


2 


Other 


0 


4 


0 


0 


2 


0 


0 


0 • 


6 


No response 


7 


35 


0 


1 


6 


0 


1 


0 


50 


Total 


61 


331 


1 


16 


26 


4 


1 


0 


440 
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TABLE 2b 



DATA FROM COLLECTION SAMPLES CONCERNING TYPE OF BINDING 



Kind of 
Sample 



Holdings 
Usage . 



Type of Binding 



















CO 




0) 














U 




bO 




bQ 


U 








a 


bO 


a 


0) 


bO 


o 




•H 


a 


•H 


x: 


a 


44 






•H 


rO t)0 


CO 


•H 


P3 


(U 


a 


no 


a a 


•H 




•H 




•H 


C 


•H *H 


fH 


a 






pq 


•H 


PQ "O 




•H 


pci 


> 




pq 


a 


;:3 


PQ 


O 


o 


44 




^ "H 


PM W) 




ctf 


o 


0) 


u 




a 


fH 


U X 


CO 


iH 


(U 


0) 


x: -H 


cj 


ctf V-i 


M ’XJ 


x: 


x: 


V-i pej 


44 




M 0) 


<U V-i 


& 




X3 


. o c 


j *H 


Xi PM 


PM ClJ. 


* S 


1 cO 


•H M 




? PM 


•H 


(0 O 


CO 


(U 


O 


O pq 


W 


hQ P4 


P4 pq 


PM 




34 


63 


0 


1 


7 


20 


0 


39 


55 


0 


Q 


2 


1 


0 



ej 

4J 

O 

H 



U 

I 

44 

o 



0 



125 



106. 



I 

i 



er|c 
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TABLE 3 a 

DATA FROM MEMORY EXPERIMENT CONCERNING HEIGHT 



Remembered 


Actual Height in Inches 




Height in 
Inches 


5 3/4 
or Less 


6- 

6 3/4 


7- 

7 3/4 


8- 

8 3/4 


9- 

9 3/4 


10- 

10 3/4 


11 or 
More 


Total 


5 3/4 or less 


0 


0 


1 


3 


0 


0 


0 


4 


6-6 3/4 . . 


0 


0 


2 


7 


3 


0 


0 


12 


7-73/4.. 


0 


0 


16 


45 


14 


0 


0 


75 


8-83/4.. 


0 


0 


4 


92 


41 


0 


0 


137 


9-93/4. . 


0 


0 


3 


36 


41 


4 


0 


84 


10 - 10 3/4 . 


0 


0 


0 


7 


14 


6 


0 


27 


11 or more . 


0 


0 


0 


7 


9 


8 


4 


28 


No response . 


0 


0 


8 


38 


24 


3 


0 


73 


Total . . 


0 


0 


34 


235 


146 


21 


4 


440 



TABLE 3b 

DATA FROM COLLECTION SAMPLES CONCERNING HEIGHT 



Kind of 
Sample 


Height in Inches 


Total 


5 3/4 
or Less 


6- 

6 3/4 


7- 

7 3/4 


8- 

8 3/4 


9- 

9 3/4 


10- 

10 3/4 


11 or 
More 


Holdings . . 


0 


0 


19 


51 


37 


2 


:,6 


125 


Usage . . . 


1 


1 


13 


47 


40 


1 


3 


106 



-204- 



TABLE 4a 

DATA FROM MEMORY EXPERIMENT CONCERNING NUMBER OF PAGES 



Remembered 
Number of 
Pages in 
Book 




Actual Number of Pages 


in Book 




Total 


0-99 


100-199 


200-299 


300-399 


400-499 


500 

or More 


0-99 .... 


11 


3 


0 


0 


0 


0 


14 


100-199 . . . 


8 


35 


26 


8 


4 


0 


81 


200-299 . . . 


0 


26 


67 


19 


16 


4 


132 


300-399 . . . 


0 


14 


32 


31 


23 


4 


104 


400-499 . . . 


0 


3 


8 


8 


13 


6 


38 


500 or more. 


0 


0 


1 


4 


16 


13 


34 


No response 


2 


8 


10 


8 


7 


2 


37 


Total . . . 


21 


89 


144 


78 


79 


29 


440 



TABLE 4b 

DATA FROM COLLECTION SAMPLES CONCERNING NUMBER OF PAGES 



Kind of 


Number of Pages in Book 




Sample 


0-99 


100-199 


200-299 


300-399 


400-499 


500 

or More 


Total 


Holdings . . . 


28 


30 


26 


20 


11 


10 


125 


Usage 


2 


21 


34 


19 


15 


15 


106 



O 
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TABLE 5a 

DATA FROM MEMORY EXPERIMENT CONCERNING CONDITION 



Remembered 

Condition 


Actual Condition 


Total 


New 


Fairly 

New 


Used 


Worn 


Bad 


New 


23 


18 


11 


4 


3 


59 


Fairly new . . . 


19 


38 


52 


13 


15 


137 


Used 


7 


12 


64 


49 


30 


162 


Worn 


1 


3 


26 


17 


17 


64 


Bad 


0 


0 


0 


0 


1 


1 


No response . . . 


1 


5 


6 


5 


0 


17 


Total . . . 


51 


76 


159 


88 


66 


440 



TABLE 5b 

DATA FROM COLLECTION SAMPLES CONCERNING CONDITION 



Kind of 


Condition 


Total 


Sample 


New 


Fairly 

New 


Used 


Worn 


Bad 


Holdings .... 


2 


11 


77 


24 


11 


125 


Usage 


13 


23 


50 


16 


4 


106 



o 
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TABLE 6a 

DATA FROM MEMORY EXPERIMENT CONCERNING DATE OF PUBLICATION 





. 




Actual Date 


of Publication 










Remembered 
Date of 
Publication 


1960 

and 

Later 


1950- 

1959 


1940- 

1949 


1930- 

1939 


1920- 

1929 


1910- 

1919 


1900- 

1909 


1890- 

1899 


1889 

and 

Before 


Other 


Total 


1960 and 
later . . 


50 


16 


3 


0 


0 


0 


0 


0 


0 


0 


69 


1950-1959 . 


37 


40 


19 


12 


1 


0 


0 


0 


0 


0 


109 


1940-1949 . 


6 


15 


16 


10 


0 


0 


2 


0 


0 


0 


49 


1930-1939 . 


1 


10 


6 


17 


6 


0 


1 


1 


0 


1 


43 


1920-1929 . 


1 


3 


2 


4 


4 


1 


0 


0 


0 


0 


15 


1910-1919 . 


1 


1 


3 


1 


1 


3 


0 


1 


0 


0 


11 


1900-1909 . 


0 


1 


0 


0 


0 


1 


1 


0 


0 


0 


3 


1890-1899 . 


0 


0 


0 


1 


0 


0 


0 


1 


0 


0 


2 


1889 and 
before 


1 


0 


0 


3 


1 


0 


0 


0 


2 


2 


9 


Other . . . 


5 


2 


1 


1 


0 


1 


1 


0 


1 


0 


12 


No response 


34 


40 


26 


10 


4 


2 


1 


0 


0 


1 


118 


Total 


136 


128 


76 

[ 


59 


17 


8 


6 


3 


3 


4 


440 



Data include non-year responses that were translated into years: 
e.g. (40's — 1945). 



TABLE 6b 



DATA FROM COLLECTION SAMPLES CONCERNING DATE OF PUBLICATION 





Date of Publication 




i\ina or 
Sample 


1960 

and 

Later 


1950- 

1959 


1940- 

1949 


1930- 

1939 


1920- 

1929 


1910- 

1919 


1900- 

1909 


1890- 

1899 


1889 

and 

Before 


Other 


Total 


Holdings 


1 

10 


! 

16 


16 


23 


31 


10 


8 


3 


8 


0 


125 


Usage , . . 


40 


25 


13 


14 


3 


3 


0 


1 


4 


3 


106 
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DATA FROM COLLECTION SAMPLES CONCERNING TYPE OF WORK 
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Total 


125 

106 


Actual Type of Work 


(aaom ao z) 


0 0 


aonuO 


<J- CO 


uofuBuaas 
-STQ ao 


0 <J- 

CO 


^pnuS asBo aiSufs 
B 30 qaoda-g 


16 1 
8 2 


iiaoda-g i^oaBasa^ 


•03a ‘SufpueqsjLapun 
- 3 T 3 S 

‘ ijuauiaAoaduiT: - 3 x a s 


CO 


auT^diDsiQ ao 
aoaCqns 30 XaoasiH 


VO 


uosaaj 
’ anoqB ^X3^TM0 
^aOjMi ao ‘^qdnaSoiq 
-oan^ *^qdBaSoig 


<N CO 


saoqany 30 
aaqmnjj b ao suq ^q 
sSuiaiaM snoiaBA 
30 uoiao^noo 


11 

26 


PT^TI B 
30 aosdsy a^SuTg B 
SuiaaAOQ qdBaSouojx 


21 

37 


aoBCqng B 
oa uoiaonpoaaui 


CO 

rH 


^[ooqaxax 


<1- to 


^ooqpuBH 


^ CO 


Kind of 
Sample 


Holdings 

Usage 
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TABLE 8a 



data from memory experiment concerning level of 

READERS FOR WHICH INTENDED 





Actual Level 


Total 


Remembered 

Level 


Popularize ticu 
1 


Treatment Under- 
standable to the 
Generally Educated 
Layman 

— H 


Work for Beginning 
College Students 


Work for Advanced , 
Students or 
Professionals 


More than One of 
the Above 


Other 


Popularization 


12 


34 


2 


4 


0 


0 


52 


Treatment understand- 
















able to the generally 
















educated layman . . . 


5 


56 


11 


57 


0 


0 


129 


Work for beginning 
















college students . . . 


0 


15 


4 


21 


0 


0 


40 


Work for advanced 
















students or pro- 
















fessionals 


0 


6 


10 


88 


0 


0 


104 


More than one of the 
















above 


5 


39 


12 


49 


0 


0 


105 


Other 


0 


2 


0 


4 


0 


0 


6 


No response 


0 


0 


1 


3 


0 


0 


4 


Total 


22 


152 


40 


226 


0 


0 


440 
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TABLE 8b 



data from collection samples concerning level 

OF READERS FOR WHICH INTENDED 



Kind of 
Sample 



Holdings 
Usage . 



a 

o 

•ri 

ra 

•H 

M 

cd 

fH 

3 

C3U 

o 



7 

0 



Level 



I 

M 

0 

X) 

g 

0) 

a 

4J 

(d 

0) 

H 



0) 

0) 4J 

^ cd 

4J O 
3 

O ^ 

4J W 

0) >> 

rO 

(d (d d 
TJ H cd 

SJ P 

td d S 
■M 0) (d 
w O 



28 

19 



bO 
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■H W 
d 4J 
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0) 



M 

0) d 
pq -M 
CO 

O 0) 
(4q bO 

M r-< 

O O 
J3 O 



32 

46 



n:d 
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•5 
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to 
■M 
d 

0) _ 
jbii 'd m 

n d o 
o -M M 

ts CO Fh 
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o 

•rl 

to 
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0) 



58 

41 
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o 



o 

0) 
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d 
td 
,d 
■M ^ 

. < 

(U 

M 0) 

o .d 
a w 



0 

0 



Total 



0) 

d 

4J 

o 



0 

0 



125 

106 



I 



er|c 
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TABLE 9a 



DATA FROM MEMORY EXPERIMENT CONCERNING INDEX 



Remembered 

Property 


Actual Property 


Total 


Index 

Present 


Index 

Absent 


Index present . . . 


155 


57 


212 


Index absent . . . 


50 


37 


87 


No response. .... 


96 


45 


141 


Total .... 


301 


139 


440 



TABLE 9b 



data from collection samples concerning index 



Kind of 
Sample 


Property 




Index 

Present 


Index 

Absent 


Total 


Holdings 


57 


68 


125 


Usage 


73 


33 


106 
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TABLE 10a 



DATA FROM MEMORY EXPERIMENT CONCERNING TABLES 





Actual 


Property 


Total 


Remembered 

Property 


Tables 

Present 


Tables 

Absent 


Tables present . . . 


48 


24 


72 


Tables absent . . . 


43 


213 


256 


No response .... 


36 


76. 


112 


Total 


127 


313 


440 . 



TABLE 10b 

DATA FROM COLLECTION SAMPLES CONCERNING TABLES 



Kind of 
Sample 


Property 




Tab les 
Present 


Tables 

Absent 


Total 


Holdings 


56 


69 


125 


Usage 


35 


71 


106 
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TABLE 11a 

DATA FROM MEMORY EXPERIMENT CONCERNING FIGURES, CHARTS, AND 7 .ttGRAMb 



Remembered 

Property 


Actual Property 




Figures, Charts, 
or Diagrams 
Present 


Figures Charts, 
or Diagrams 
Absent 


Total 


Figures, charts, or 

diagrams present .... 


59 


26 


85 


Figures, charts, or 

diagrams absent .... 


29 


208 


237 


No response 


48 


70 


118 


Total 


136 


304 


440 



TABLE 11b 

DATA FROM COLLECTION SAMPLES CONCERNING 
figures, charts, AND. DIAGRAMS 





Property 




Kind of 
Sample 


Figures, Charts, 
or Diagrams 
Present 


Figures, Charts 
or Diagrams 
Absent 


Tota 1 


Holdings 


43 


82 


125 


Usage ... 


36 


70 


106 
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TABLE 12a 

DATA FROM MEMORY EXPERIMENT CONCERNING 
CHAPTER TITLES 



Remembered 

Property 


*» 

•Actual Property 


Total 


Chapter 

Titles 

Present 


Chapter 

Titles 

Absent 


Chapter titles present . 


350 


10 


360 


Chapter titles absent 


22 


7 


29 


No response 


49 


2 


51 


Total 


421 


19 


440 



TABLE 12b 

DATA FROM COLLECTION SAMPLES CONCERNING 
CHAPTER TITLES 





Property 




Kind of 
Sample 


Chapter 

Titles 

Present 


Chapter 

Titles 

Absent 


Total 


Holdings 


89 


36 


125 


Usage 


93 


13 


106 
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TABLE 13a 

data from memory experiment concerning 
quotations from literary works 





Actual Property 


Total 


Remembered 

Property 


Quotations 

Present 


Quotations 

Absent 


Quotations present . . 


52 


32 


84 


Quotations absent . . 


35 


126 


161 


No response 


55 


140 


195 


Xo1I£LX ##••••* 


142 


298 


440 



TABLE 13b 

data from collection samples concerning 
quotations from literary works 





Property 


Total 


Kind of 
Sample 


Quotations 

Present 


Quotations 

Absent 


Holdings 


40 


85 


125 


Usage 


52 


54 


106 



f 



1 
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TABLE 14a 

DATA FROM MEMORY EXPERIMENT CONCERNING 
CASE STUDIES 



Remembered 

Property 


Actual Property 


Total 


Case 

Studies 

Present 


Case 

Studies 

Absent 


Case studies present . . 


119 


70 


189 


Case studies absent . . 


17 


120 


137 


No response 


30 


84 


114 


Total 


166 


274 


440 



TABLE 14b 

data from collection samples concerning 

CASE STUDIES 





Property 




Kind ol; 
Sample 


Case 

Studies 

Present 


Case 

Studies 

Absent 


Total 


Holdings 


27 


98 


125 


Osd'^6 •• ii •••••• 


25 


81 


106 







< 
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TABLE 15a 

DATA FROM MEMORY EXPERIMENT CONCERNING GRAPHS 



Remembered 

Property 


Actual Property 


Total 


Graphs 

Present 


Graphs 

Absent 


Graphs present .... 


24 


25 


49 


Graphs absent .... 


25 


243 


268 


No response 


39 


84 


123 


Toficil •••••■ 


88 


352 


440 



TABLE 15b 

DATA FROM COLLECTION SAMPLES CONCERNING GRAPHS 



Kind of 
Sample 


Property 




Graphs 

Present 


Graphs 

Absent 


Total 


Holdings . . . . 




37 


88 


125 


Usage 




20 


86 


106 



r 
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TABLE 16a 

DATA FROM MEMORY EXPERIMENT CONCERNING 
PREFATORY MATERIAL 





Actual Property 


Total 


Remembered 

Property 


Preface 

Present 


Preface 

Absent 


Preface present . . . 


240 


16 


256 


Preface absent . , . 


20 


7 


27 


No response 


143 


14 


157 


Total 


403 


uJL- 


440 



TABLE 16b 

DATA FROM COLLECTION SAMPLES CONCERNING 
PREFATORY MATERIAL 





Prop 


erty 


Total 


Kind of 
Sample 


Preface 

Present 


Preface 

Absent 


Holdings 


87 


38 


125 


Usage 


84 


22 


106 



o 
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TABLE 17a 

DATA FROM MEMORY EXPERIMENT CONCERNING FOOTNOTES 



Remembered 

Property 


Actual Property 


Total 


Footnotes 

Present 


Footnotes 

Absent 


Footnotes present . . 


122 


29 


151 


Footnotes absent . . 


79 


59 


138 


No response 


105 


46 


151 


Total 


306 


134 


440 



table 17b 

DATA FROM COLLECTION SAMPLES CONCERNING FOOTNOTES 



Kind of 
Sample 


Property 


Total 


Footnotes 

Present 


Footnotes 

Absent 


Holding 


75 


50 


125 


Usage 


69 


37 


106 
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TABLE 18a 



DATA FROM MEMORY EXPERIMENT CONCERNING 
BIBLIOGRAPHY OR SUGGESTED READINGS 



Remembered 

Property 


Actual Property 




Bibliography 

Present 


Bibliography 

Absent 


Total 


Bibliography present . 


76 


66 


142 


Bibliography absent 


42 


66 


108 


No response 


100 


90 


190 


Total 


218 


222 


440 



TABLE 18b 

DATA FROM COLLECTION SAMPLES CONCERNING 
BIBLIOGRAPHY OR SUGGESTED READINGS 



Kind of 
Sample 


Property 


Total 


Bibliography 

Present 


Bibliography 

Absent 


Holdings 


64 


61 


125 


Usage 


61 


45 


106 
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TABLE 19a 

data from memory experiment concerning dedication 



Remembered 

Property 


Actual Property 


Total 


Dedication 

Present 


Dedication 

Absent 


Dedication present . . 


38 


33 


71 


Dedication absent . . 


17 


41 


58 


No response 


141 


170 


311 


Total 


196 


244 


440 



TABLE 19b 



data from collection samples concerning DEDICATION 



Kind of 
Sample 


Property 


Total 


Dedication 

Present 


Dedication 

Absent 


Holdings 


29 


96 


125 


USd^B *•••••*• 


27 


79 


106 
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TABLE 20a 



DATA FROM MEMORY EXPERIMENT CONCERNING GLOSSARY 



Remembered 

Property 


Actual Property 


Total 


Glossary 

Present 


Glossary 

Absent 


Glossary present . . . 


4 


21 


25 


Glossary absent . . . 


7 


240 


247 


No response 


10 


158 


168 


Total 


21 


419 


440 



table 20b 



DATA FROM COLLECTION SAMPLES CONCERNING GLOSSARY 



Kind of 
Sample 


Property 


Total 


Glossary 

Present 


Glossary 

Absent 


Holdings . . . . 




4 


121 


125 


Usage . . . . . 




7 


99 


106 
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TABLE 21a 

data from memory experiment concerning problems 

AT ENDS OF CHAPTERS 





Actual Property 


Total 


Remembered 

Property 


Problems 

Present 


Problems 

Absent 


Problems present . . . 


1 


12 


13 


Problems absent . . • 


3 


357 


360 


No response 


2 


65 


67 


Total 


6 


434 


440 



TABLE 21b 

data from COLLECTION SAMPLES CONCERNING 
PROBLEMS AT ENDS OF CHAPTERS 





Property 


Total 


Kind of 
Sample 


Problems 

Present 


Problems 

Absent 


Holdings 


4 


121 


125 


Usage 


2 


104 


106 



7 
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TABLE 22a 

DATA FROM MEMORY EXPERIMENT CONCERNING 
SINGLE -VOLUME WORKS 





Actual Property 




Remembered 

Property 


Single- 

Volume 

Work 


Not a 
Single- 
Volume 
Work 


Total 


Single-volume work » . 


411 


2 


413 


Not a single-volume 
work 


9^ 


8 


17 


No response 


8 


2 


10 


Total 


428 


12 


440 



TABLE 22b 

DATA FROM COLLECTION SAMPLES CONCERNING 
SINGLE -VOLUME WORKS 





Property 




Kind of 
Sample 


Single- 

Volume 

Work 


Not a 
Single- 
Volume 
Work 


Total 


Holdings 


121 


4 


125 


Usage » 


100 


6 


106 



O 
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TABLE 23a 



DATA FROM MEMORY EXPERIMENT CONCERNING 
TRANSLATED WORK.S 





Actual Property 




Remembered 

Property 


Translated 

Work 


Not a 

Transalted 

Work 


Total 


Translated work . . . 


21 


12 


33 


Not a translated 

work 


25 


267 


292 


No response 


14 


101 


115 


Total 


60. 


380 


440 



TABLE 23b 

f, 

DATA FROM COLLECTION SAMPLES CONCERNING 

translated works 





Property 




Kind of 
Sample 


Translated 

Work 


Not a 

Translated 

Work 


Total 


Holdings 


12 


113 


125 


Usage . . 


20 


86 


106 



i 






o 



ERIC 

MMilRlffriTlMJ 
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table 24a 

data from memory experiment concerning 

REPRINTED OR REVISED WORKS 





Actual Property 




Remembered 

Property 


Reprinted 

or 

Revised 


Not 

Reprinted 
or Revised 


Total 


Reprinted or revised , 


16 


8 


24 


Not reprinted or 

revised • 


32 


183 


215 


No response ..... 


36 


165 


201 




84 


356 


440 



TABLE 24b 



DATA FROM COLLECTION SAMPLES CONCERNING 
REPRINTED OR REVISED WORKS 





Property 




Kind of 
Sample 


Reprinted 

or 

Revised 


Not 

Reprinted 
or Revised 


Total 


Holdings 


17 


108 


125 


Usage 


13 


93 


106 



